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SECTION  1. 


EVALUATION  OVERVIEW 

1.1  Background.  The  Navy  has  little  experience  in  evaluating 
the  usefulness  of  numerical  environmental  predictions  prepared 
for  periods  greater  than  3  days.  Recognizing  this,  the  Naval 
Environmental  Prediction  Research  Facility  ( NEPRF )  embarked  in 
1983  on  a  four-part;  project  to  develop  a  procedure  which  could 
establish  the  relevant  accuracy  and  operational  utility  of  a 
medium-range  (5  to  10  day)  atmospheric  forecast  system.  J 

In  the  first  part  of  this  Medium- Range  Forecast  Evaluation 
( MRFE )  project,  a  review  was  prepared  which  addressed  the  present 
accuracy  and  operational  use  of  medium-range  numerical  forecasts 
for,  in  particular.  Navy  applications.  This  review  (Elsberry, 
Hamilton  and  Petit,  1984)  describes  present  levels  of  medium- 
range  forecast  skill  and  sets  forth  acceptable  medium-range 
levels  of  accuracy  for  various  operationally  relevant  weather 
parameters . 

In  the  second  part  of  this  MRFE  project,  a  detailed  plan  for 
evaluating  the  likely  operational  benefits  of  a  medium-range 
forecast  system  was  prepared.  The  Medium-Range  Atmospheric 
Forecast  System  Evaluation  Plan  (Petit,  Hamilton  and  Elsberry, 
1984)  calls  for  two  months  of  forecast  data  collection  and 
subsequent  verification  and  assessment  of  skill  and  utility  of 
both  objective  and  subjective  forecasts. 

The  third  part  of  this  MRFE  project  consisted  of  collecting 
and  saving  selected  output  from  the  Navy  Operational  Global 
Atmospheric  Prediction  Systm  (NOGAPS)  and  various  forecasts 
prepared  therefrom  in  accordance  with  the  Evaluation  Plan.  This 
collection  process  commenced,  depending  upon  the  type  of 
evalution'  involved,  on  or  about  1  April  1984.  An  extended 
version  of  NOGAPS  was  run  off-time  on  the  operational  computers 
at  the  Fleet  Numerical  Oceanography  Center  (FNOC),  Monterey, 
California.  The  data  collection  required  the  design  and  regular 


execution  of 


on 


the 


FNOC 


several  comolex  comouter  orocedures 


computers 


The  fourth  part  of  this  MRFE  project  consists  of  evaluating 
the  data  saved  during  the  collection  phase  (part  three)  and 
reporting  on  the  MRFE  procedures  developed  and  the  accuracy  and 
utility  of  the  NOGAPS  for  medium-range  forecasts.  This  report  is 
the  required  part  four  documentation. 

The  data  reduction  for  part  four  of  the  project  required 
that  substantial  new  or  greatly  modified  computer  software  be 
prepared.  The  resultant  procedures  and  computer  programs  have 
been  turned  over  to  NEPRF  for  use  in  subsequent  evaluations. 

1.2  Period  of  Evaluation.  The  NOGAPS  forecast  fields,  derived 
fields  and  special  forecasts  and  their  verifying  analysis  fields 
and  synoptic  reports  were  collected  during  the  period  24  March 
through  15  June  1984.  All  MRFE  associated  data  were  prepared  and 
collected  once  daily  for  the  00  GMT  synoptic  time,  since  no 
extended  forecasts  were  prepared  with  the  12  GMT  meteorological 
data.  Details  of  this  collection  effort  were  reported  in  SASC 
(1984).  In  summary: 

Field  data  collection  (forecast  and  verifying  analyses) 
commenced  on  24  March  and  continued  through  31  May,  but  only 
30  to  35  actual  days  of  forecasts  (depending  upon  product 
and  length  of  forecast  (TAU))  were  prepared,  saved  and 
verified.  (See  subsection  1.3  for  a  description  of  certain 
problems  encountered.) 

Five-day  mean  storm  track  calculations  from  the 
standard  fields  began  on  28  March  and  continued  through  15 
June,  and  57  fields  were  successfully  verified. 


*< 


Area  wind  warning  calculations  began  on  25  March  and 
continued  through  15  June.  Of  the  resulting  wind  warning 


assessment  sets,  53  for  five  days  and  51  for  seven  days  were 
successfully  verified. 

Sensible  weather  forecasting  on  a  twice  weekly  basis 
(when  possible)  commenced  12  April  and  continued  through  9 
June.  Sixteen  5  and  7  day  forecasts  were  prepared  by  each 
of  four  forecasters  and  all  were  verified  with  observations 
collected  between  17  April  and  15  June. 

Systematic  Error  Identification  System  (SEIS)  data 
collection  began  on  10  April  and  continued  through  15  June, 
but  only  30  SEIS  runs  were  successfully  completed  during 
this  62  day  period. 

Five-day  mean  storm  track  calculations  using  SEIS 
output  were  never  prepared  as  planned,  as  explained  in 
subsection  2. 2. 1.1. 

1.3  Problems  Encountered.  As  discussed  in  subsection  1.2,  half 
or  more  of  the  potential  f orecasfc/ver i f ication  pairs  in  certain 
categories  were  missed.  A  sixty-plus  day  evaluation  period 
succeeded  in  collecting  only  about  one  month's  data.  This  is 
attributed  to  the  following: 

NOGAPS  Fields  Not  Available.  NOGAPS  fields  were  not 
prepared  on  30  March,  on  9  and  16  April,  on  3,  4,  5,  6,  8, 
15,  22,  and  29  May  and  on  1  and  7  June  (13  days  in  all).  On 
these  days  analyses  were  used  to  verify  previous  forecasts, 
but  the  missing  forecasts  were  unrecoverable  and  affected 
products  involving  adjacent  days  (for  example,  lagged 
averages) .  The  five  days  missed  in  early  May  were 
apparently  associated  with  installing  a  major  forecast  model 
change.  The  reasons  for  other  missed  runs  included  a  power 
outage,  late  start-early  termination,  operational 
priorities,  etc. 
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NOGAPS  Model  Change.  As  mentioned  above,  in  early  May  a 
m.a jor  change  to  the  forecast  model  was  installed  -  not 
without  difficulty.  This  model  change  was  not  well 
documented,  but  apparently  affected  only  the  day  6  and  day  7 
(TAU  144  and  163)  output.  Besides  the  trauma  of 
implementing  the  change,  it  also  makes  detailed  performance 
statistics  and  month- to-month  comparisons  somewhat  suspect. 

Magnetic  Tape  Difficulties.  Rotating  mass  storage  was  not 
available  for  storing  the  many  fields  and  raw  verification 
data  required  for  this  evaluation.  All  such  information  was 
routinely  written  to  magnetic  tape  for  further  processing  or 
for  backup  purposes.  Many  frustrations  resulted.  First, 
there  was  a  critical  shortage  of  magnetic  tapes  at  Fleet 
Numerical  Oceanography  Center  (FNOC)  and  it  became  necessary 
to  stack  several  days  data  onto  one  tape  rather  than  keep 
days  separated  as  planned.  Similarly,  multi-day  collections 
of  verification  data  had  to  be  stacked.  This  complex 
process  caused  one  serious  procedural  error  or  oversight 
which  resulted  in  the  loss  of  field  verification  statistics 
for  10  days  during  late  April  and  early  May.  Second,  at 
least  one  physical  tape  unit  at  FNOC  proved  highly 
unreliable  and  there  were  difficulties  with  some  pretested 
reels  of  tape.  Most  of  the  resulting  problems  were 
corrected,  but  two  complete  days  of  basic  forecast  fields 
proved  unrecoverable  (2  and  11  April). 


Cancelled  or  Delayed  Jobs .  Because  the  six  and  seven  day 
forecast  fields  were  not  being  produced  for  operational  use 
and  because  all  of  the  derivations  and  verifications  were  of 
low  execution  priority,  there  were  frequent  cases  of  late 
field  data  availability  and  of  verification  jobs  being 
"locked  out"  for  extended  periods.  In  some  instances  they 
were  simply  dropped  by  the  operator  in  favor  of  higher 
priority  and/or  classified  processing.  In  a  very  few  cases, 
individual  derived  products  or  verifications  were  never  made 


SECTION  2 


FINDINGS 


2.1  Field  7 e ! 


ica: 


:ns 


The  field  verification  or o cede res  are 


described  in  subsection  2.2.1  of  the  Evaluation  Plan.  Global 
forecast  fields  of  height,  temperature  and  wind  (u  and 
components)  at  four  levels  (1000,  350,  500  and  200  mo)  for  four 

TAU ' s  (96,  120,  144  and  168  hours)  were  verified.  The  1000  and 

500  mb  fields  were  also  used  to  derive  the  non-standard  fields 
specified  in  Figure  2-01. 


In  accordance  with  the  Evaluation  Plan,  field  verifications 
were  made  and  analyzed  for  April  and  May  and  for  both  months 
combined,  for  each  of  forecast  days  4  through  7  (TAG'S  95  through 
163)  in  terms  of  several  objective  scores  for  the  five  major 
areas  and  seven  subareas  listed  in  Table  1. 

TABLE  1.  Verification  Areas 

NR.  OF  STD  . 


AREA 

COORDINATES 

GRID  ?0 

Northern  Hemisphere 

20N-30N, 60E-0-57. 5E 

3600 

Tropics 

20N-20S,60E-0-57. 5E 

244  3 

Tropical  N.  Atlantic 

00N-20N , 20-90W 

261 

Tropical  W.  Pacific 

20N-20S , 100E-180 

1241 

North  Indian  Ocean 

00N-20N, 50-100E 

153 

Southern  Hemisphere 

20S-80S, 60E-0-57. 5E 

3600 

North  Pacific 

70N-20N, 120E-120W 

10  2  9 

Northwest  Pacific 

70N-20N, 120E-180 

525 

Northeast  Pacific 

70N-20N,  180-120W 

525 

North  Atlantic 

70N-20N, 80W-10E 

777 

Northwest  Atlantic 

70N-20N, 8  0W-3  5W 

399 

Northeast  Atlantic 

70N-20N, 35W-10E 

399 

The  North  Pacific  area  approximates  the  Storm  Track  and  SETS 
areas.  For  area  means  (see  below),  a  cosine  weighting  function 
was  used  to  compensate  for  the  decrease  in  area  as  a  function  of 
latitude  from  the  equator  to  either  pole. 
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Ternueracure  Ranae 


Time  Wind  Direction  Range 


Time  Wind  Magnitude  Range 


Lagged  Temperature  Range 


Lagged  Wind  Direction  Range 


Lagged  Wind  Magnitude  Range 


Spectral  Height  Truncation  Waves  1 


ectral  Height  Truncation  Waves  4  thru  9 


Time  Average  Height  All  Waves 


Time  Average  Height  Trunc .  Waves  1  thru  3 


Time  Average  Height  Trunc.  Waves  4  thru  9 


Lagged  Average  Height  All  Waves 


Lagged  Average  Height  Trunc.  Waves  1  thru  3 


Lagged  Average  Height  Trunc.  Waves  4  thru  9 


Time  Temperature  Range 


Time  Wind  Direction  Range 


Time  Wind  Magnitude  Range 


Lagged  Temperature  Range 


Lagged  Wind  Direction  Range 


Lagged  Wind  Magnitude  Range 


derived  for  use  as  a  verifying  analysis, 
derived  from  verification. 

derived  from  the  same  calendar  day  and  time  data  from  ‘ 
most  recent  daily  forecast  runs. 

derived  from  the  three  TAUs  indicated,  all  from  same  r- 


1GURE  2-01.  Derived  Eiei 
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Except  as  noted,  the  following  objective 
computed  for  the  basic  fields  for  ail  areas, 
variables: 

•  mean  error  of  forecast  (only  the  wind  in  the  tropics) 

•  mean  error  of  persistence  (wind  only,  tropics  only) 

•  root-mean-square  error  of  forecast  (only  the  wind  in  the 
tropics) 

•  root-mean-square  error  of  persistence  (wind  only,  tropics 
only) 

•  standard  deviation  of  forecast  error  (only  the  wind  in  the 
tropics ) 

•  standard  deviation  of  persistence  error  (wind  only,  tropics 
only) 

•  standard  deviation  of  verifying  anomaly  (heights  and 
temperatures  only,  not  in  tropics) 

•  anomaly  correlation  of  forecast  (heights  and  temperatures 
only,  not  in  tropics) 

These  scores  were  computed  using  the  following  expressions: 


1/n  Z  ( F-A v )  =  ( F -A v )  =  mean  error  of  forecast 
1/n  Z  (Aq-av)  =  ( AQ-AV )  =  mean  error  of  persistence 

/  p" 

1/n  E  (F-Av)  =  rmse  of  forecast 

— 

*  1/n  Z  (A_-A„)  =  rmse  of  persistence 


3 


1/n  Z  [  (Aq-Av)  -  (Aq-Av)  ]  *•  =  standard  deviation  of 

persistence  error 


1/n  E  [ (Av-C ) - (A  -C) ] 1  =  standard  deviation  of  verifying 


anomaly 


{  [  (  F-C  )  -  ( F-C  )  ]  [  ( A..-C  )  -  ( A,,-C  )  ]  }  =  anomaly  correlation 


[  (F-C)-  (F-G) ] z  [  (Av-C)-  (Av-C; 


for  forecast 


where : 


Aq  =  initial  analysis 
Av  =  verifying  analysis 
F  =  forecast 
C  =  monthly  climatology 

n  =  number  of  gridpoints  in  the  verification  area 

F-C  =  predicted  anomaly 

Av-C  =  verifying  anomaly 

F-Av  =  forecast  error 

Aq-Av  =  persistence  error 

(overbar)  =  area  mean 

Vector  wind  (  v)  errors  were  calculated  in  wind  component  form  as 
follows : 


mean  error  V  =  [mean  error (u) ]  +  [mean  error  (v) ] ^ 

rmse(V)  =  [rmse(u)]2  +  [rmse(v)]2 


[stdv(u)]2  +  [stdv(v)]2 


stdv (V) 


Vector  wind  errors  were  ca.cu.acec  ac  230  mu  for  been 
and  the  tropics.  Scalar  wind  direction  and  magni 
were  separately  calculated  for  all  areas  and  levels. 

In  the  tropics  as  a  whole  and  in  the  three  tropical  subareas 
separately,  the  ability  to  forecast  substantial  change  in  vector 
wind  at  1000  and  200  mb  was  assessed  at  each  grid  point. 
Substantial  change  thresholds  were  set  at  10  and  25  kt  for  the 
two  levels  respectively.  Contingency  tables  were  then  be  used  to 
compute  error  reduction  and  forecast  bias  statistics. 

The  derived  fields  listed  in  Figure  2-01  were  verified  for 
the  two  hemispheres  and  two  ocean  basins  only.  These 
verifications  were  in  terms  of  mean  error  of  forecast,  rmse  of 
forecast  and  standard  deviation  of  forecast  error  for  the 
spectral  truncations  and  averages.  The  derived  range  fields  were 
verified  in  terms  of  percent  of  points  within  the  specified  range 
at  00  GMT  on  Day  5. 

The  complete  April,  May  and  two-month  combined  verification 
scores  were  provided  to  NEPRF  under  separate  cover.  These  scores 
are  discussed  in  summary  form  by  level  and  parameter  in 
subsequent  subsections. 

2.1.1  Basic  Field  Skills. 

2. 1.1.1  1000  Millibar  Level. 

2. 1.1. 1.1  Heights.  Mean  1000  mb  height  errors,  which  are 
generally  positive  for  most  times  and  TAU's,  range  from  less  than 
9  m  at  T96  (TACJ  96)  to  less  than  23  m  at  T168.  Exceptions  are 
the  consistently  negative  errors  in  the  April  W.  PAC  (Western 
Pacific),  April  S.  HEMI  (Southern  Hemisphere)  and  May  W.  ATL 
(Western  Atlantic).  The  very  high  positive  (20-36  m)  mean  errors 
in  the  May  S.  HEMI  are  also  noteworthy.  RMS  (Root-Mean-Square) 
errors  at  1000  mb  range  from  45  m  (May,  E.  PAC,  T96)  to  117  m 


(May,  3.  HEM  I  ,  T163),  but  are  generally  53-75  m.  AC  (anomaly 
correlation)  scores  range  from  3.34  (May,  W.  ACL,  C163)  to  a  high 
of  3.65  (May,  E.  ATL,  TI23),  but  are  generally  3.53-3.53  at  C96 
and  3.23-3.30  at  T163.  There  is  generally  a  correlation  between 
low  RMS  and  high  AC  values.  There  is  no  apparent  correlation 
between  the  magnitude  of  the  verifying  anomaly  and  either  the  RMS 
or  the  AC. 

2. 1.1. 1.2  Temperatures.  Mean  1030  mb  temperature  errors  are 
generally  negative  and  less  than  1  deg  C.  Exceptions  are  the 
small  positive  errors  during  April  in  W.  ATL  and  during  both 
months  in  S.  HEM I .  Larger  (nearly  2  degrees)  positive  mean 
errors  appear  only  during  April  in  W.  PAC  at  T144  and  T163.  RMS 
errors  range  from  3.3  deg  (May,  W.  PAC,  T96)  to'  10.0  deg  (April, 
W.  ATL,  T 16 8 ) .  AC  scores  range  from  0.12  (May,  E.  PAC,  T168)  to 
0.58  (May,  N.  HEMI ,  T96).  There  is  reasonable  correlation 
between  low  RMS  and  high  AC  values,  but  no  apparent  correlation 
between  either  score  and  the  magnitude  of  the  verifying  anomaly. 

2 .  1 . 1 . 1 .  3  Winds .  The  directional  errors  of  wind  at  1030  mb  are 
large  in  all  areas.  RMS  errors  are  generally  between  70  and  75 
deg  at  the  lower  TAU '  s  and  between  80  and  85  deg  at  TAUs  144  and 
168.  They  are  5  to  10  deg  higher  in  the  tropics  where 
persistence  provided  a  slightly  (5-9  deg  RMS)  better  wind 
direction  forecast  than  did  NOGAPS.  The  small  mean  errors  of 
direction,  generally  negative  in  April  and  positive  in  May,  are 
not  considered  significant. 

The  mean  scalar  wind  speed  errors  are  generally  less  than  2 
kt  and  are  positive  in  April  and  negative  in  May.  RMS  speed 
errors  at  1000  mb  are  mostly  between  7  and  10  kts,  slightly 
higher  (10-11  kts)  in  the  S.  HEMI  and  only  slightly  less  at  lower 
TAU ' s  than  at  higher  TAU's.  Again,  persistence  provided  a 
slightly  (0.3-0. 8  kt  RMS)  better  wind  speed  forecast  than  NOGAPS, 
but  such  a  small  difference  may  not  be  statistically  significant. 
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substantial  (13  :<t)  changes  in  13-3  3  mb  wind  speed  in  the  tropics 
were  incorrect  and  the  corresponding  statistics  in  the  Appendix 
should  be  ignored.  (The  200  mo  values  are  correct  and  will 
subsequently  be  discussed.) 

2. 1.1. 2  850  mb  Level. 

2. 1.1. 2.1  Heights.  Mean  850  mb  height  errors  are  generally 
positive  and  range  from  up  to  11  m  at  T96  to  up  to  20  m  at  T168. 
As  in  the  case  of  1000  mb  heights,  there  are  much  larger  positive 
(20-37  m)  mean  errors  in  the  May  S.  HEMI.  Exceptions  are  the 
consistently  negative  errors  in  the  April  and  May  W.  PAC  and  the 
May  W.  LANT.  RMS  errors  at  850  mb  range  from  39  m  (May,  E.  PAC, 
T96)  to  110  m  (May,  S.  HEMI,  T168),  but  are  generally  50  -  75  m. 
AC  scores  range  from  0.18  (May,  E.  PAC,  T168)  to  0.67  (May,  E. 
LANT,  T 120  (same  as  1000  mb)),  but  are  generally  0.50-0.60  at  T96 


and  near  0.30  at  T168 


There  is  a  reasonable  correlation  between 


low  RMS  and  high  AC,  but  no  correlation  between  either  of  these 
and  the  magnitude  of  the  anomaly. 


2. 1.1. 2. 2  Temperatures.  Mean  temperature  errors  at  850  mb  are 
generally  negative  and  and  1  deg  or  less  for  the  two  months 
combined,  but  show  considerable  month- to-month  variation.  They 
vary  from  less  than  0.2  deg  C  for  April  in  the  N.  HEMI  to  1.6  deg 
or  more  in  May.  The  opposite  is  true  in  the  S.  HEMI  where  in 
April  the  mean  error  is  in  excess  of  1  deg  at  all  TAU's,  while  in 
May  the  errors  lie  between  0.35  and  0.7  deg.  RMS  temperature 
errors  are  slightly  higher  in  both  hemispheres  in  May.  They 
range  from  3.1  (May,  E.  LANT,  T96)  to  6.2  deg  (April,  E.  PAC, 
T168).  AC  scores  are  in  accord  by  being  substantially  lower  in 
both  hemispheres  in  May  at  ail  TAU's.  They  range  from  3.11  (May, 
W.  LANT,  T 1 63 )  to  0.68  (April,  N.  HEMI,  T96).  The  N.  HEMI 
anomalies  are  higher  in  April,  when  AC  scores  are  high,  than  in 
May  when  AC  scores  are  lower.  The  opposite  is  true  in  the  S. 


HEMI.  This  lack  o f  any  consistent  correlation  between  the 
cagni tude  o f  the  anomalies  and  the  ocher  scores  will  no c  oe 
commented  upon  further  i n  the  f ol lowing  subsections  . 

2 . 1 . 1 . 2  .  3  Winds-.  The  mean  directional  errors  of  wind  at  3  53  mb 
are  large  in  all  areas  except  the  tropics.  They  are  slightly 
less  than  at  1000  mb.  RMS  errors  are  generally  between  73  and  75 
deg  at  the  lower  TAU's  and  between  75  and  33  deg  at  T144  and 
T168.  The  RMS  errors  are  over  90  deg  at  all  TAU's  in  the  tropics 
where  persistence  again  provided  the  better  forecast  by  about  5 
deg  RMS. 

Although  mean  wind  speed  errors  are  generally  positive  and 
less  than  2  kt  in  April,  they  are  very  decidedly  negative  in  May, 
when  mean  speed  errors  are  over  three  knots  at  higher  TAU's.  RMS 
speed  errors  are  mostly  between  10  and  13  kts,  slightly  higher  in 
the  S.  HEMI  and  only  a  knot  or  two  less  at  T96  than  at  T168. 
Again,  persistence  provided  a  fractionally  better  overall 
forecast  than  MOGAPS  at  these  time  scales. 

2. 1.1. 3  500  mb  Level. 


2. 1.1. 3.1  Heights.  Mean  height  errors  at  500  mb  are  generally 
positive,  especially  in  April.  They  range  from  a  low  of  about  -1 
m  to  nearly  70  m.  RMS  errors  range  from  52  m  (May,  E.  PAC ,  T96) 
to  147  m  (May,  S.  HEMI,  T168)  but  most  of  the  errors  are 
generally  between  75  and  115  m.  AC  scores  range  from  3.07  (May, 
W.  ATL,  T168)  to  0.77  (May,  ATL ,  T96),  but  are  generally  0.60- 
0.70  at  T96  and  0.30-0.40  at  T168.  AC  scores  are  nearly  always 
lower  in  the  S.  HEMI  than  in  the  N.  HEMI.  There  is  the  usual 
correlation  between  RMS  and  AC  scores. 

2. 1.1.3. 2  Temperatures.  Mean  temperature  errors  at  530  mb  are 
positive  for  both  months,  and  all  TAU's  and  levels.  They  range 
from  1-2  deg  at  T96  to  2-3  deg  at  T168,  although  they  are 
slightly  higher  in  the  5.  HEMI.  RMS  errors  range  from  2.9  (May, 
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E.  PAC,  T  9  6  )  to  5.5  (May,  5.  HEMI  ,  T 1 6  3 )  .  AC  scores  tor  533  me 
temperatures  range  from  3.33  (May,  E.  PAC,  T96)  to  3.52  (April, 
E.  ATL,  T96).  There  is  the  customary  correlation  between  higher 
RMS  and  lower  AC  scores. 


2 . 1 . 1 . 3 . 3  Winds . 


The  directional  errors  of  wind  at  533  mb  lie 


between  55  and  73  and  are  about  15  degrees  less  than  at  the  lower 
levels.  The  exception  is  the  tropics  where  they  remain  between 
83-90  deg  and  are  essentially  identical  to  the  error  of 
persistence  at  all  TAU's. 

The  wind  speed  mean  errors  are  slightly  more  negative  in  May 
than  in  April  and  range  between  0  to  4  kt  in  the  former  and  2-6 
kt  in  the  latter.  RMS  speed  errors  are  15-20  kt  except  for  the 
tropics  where  they  are  8-10  kt,  which  is  slightly  less  than  the 
persistence  forecast  for  both  months  at  all  TAU's. 

2. 1.1. 4  200  mb  Level. 

2.1, 1.4.1  Heights.  Mean  200  mb  height  errors  are  all  positive, 
are  much  larger  than  at  500  mb,  and  vary  little  from  April  to 
May.  They  range  from  41  m  (April,  N.  HEMI,  T96)  to  151  m  (May, 
S.  HEMI,  T168).  RMS  errors  are  somewhat  higher  in  April  than  in 
May  except  in  the  S.  HEMI.  They  range  from  98  m  (May,  E.  PAC, 
T96)  to  237  m  (May,  S.  HEMI,  T168),  but  are  generally  110-193  m. 
AC  scores  range  from  0.15  (May,  E.  PAC,  T168)  to  0.75  (May,  E. 
ATL,  T 96 ) ,  but  are  generally  0.60-3.70  at  T96  and  0.25-0.40  at 
T 168 .  As  expected,  the  AC  scores  at  all  TAU's  are  lower  in  the 


S.  HEMI  than  in  the  N.  HEMI. 


There  is  the  usual  loose 


correlation  between  RMS  and  AC  scores. 

2. 1.1. 4. 2  Temperatures.  Mean  temperature  errors  at  200  mb  are 
generally  negative  0.5-1. 5  deg.  Two  exceptions  are  the  E.  ATL  in 
April  and  the  W.  ATL  in  May  where  they  were  less  than  1  deg  but 
positive.  RMS  errors  range  from  4-5  deg  at  T96  to  5-7  deg  at 
T163,  which  is  a  degree  or  so  higher  than  at  500  mb.  The 
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the  same  area  (5.  ATL)  with  a  low  of  -3.33  'May,  7153  )  and  a  hi  on 
of  3.60  (April,  T96).  The  correlation  between  higher  RMS  and 
lower  AC  is  again  noted. 

2.1. 1.4. 3  Winds.  The  230  mb  directional  errors  lie  between  43 
and  65  and  are  5-10  degrees  less  than  at  500  mb.  The  exception 
is  again  in  the  tropics,  where  they  remain  near  83  deg  and  are 
about  5  deg  worse  (higher)  than  persistence  at  all  TAU's. 

The  mean  wind  speed  errors  are  slightly  more  negative  in  May 
than  in  April.  They  range  between  +2  and  -8  in  May  and  between 
-15  and  -7  in  April.  They  are  most  negative  in  the  tropics  in 
both  months.  RMS  speed  errors  at  200  mb  are  22-27  kt  at  the 
lower  TAU's  and  28-33  kt  at  the  higher  TAU's;  except  in  the 
tropics  when  they  are  5-10  kt  less  than  in  other  areas  and 
essentially  equal  to  persistence  (less  than  +1  kt). 


Vector  winds  errors  were  computed  at  233  mb.  They  lie 
between  37  kt  (T96)  and  44  kt  (T168)  in  the  N.  HEMI. 
Corresponding  figures  for  the  S.  HEMI  are  43  and  52  kt,  and  for 
the  tropics  they  are  32  and  34  kt.  The  vector  errors  of 
persistence  in  the  tropics  were  40-41,  so  the  model  was  clearly 
better  than  persistence  in  this  comparison. 

The  skill  of  the  model  in  forecasting  substantial  (25  kt  or 
more)  changes  in  wind  speed  at  200  mb  in  the  tropics  and  its 
three  subareas  was  also  assessed.  Contingency  tables  were  used 
to  compute  the  error  reduction  with  respect  to  the  chance  of  the 
event  occuring  in  nature  and  to  compute  the  bias  with  respect  to 
over f orecas t i ng  the  25  kt  wind  speed  change  event  (positive  bias) 
or  underforecasting  the  event  (negative  bias).  The  results  are 
oresented  in  Table  2. 
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Month 


April 


Total 


TABLE  2.  Skill  in 

230  mb. 

Forecast! 

ng  Change  ( >  2  5  kt)  a 

Error  Reduction/Bias 

Area 

T  9  6 

T120 

T144 

T163 

Tropics  (all) 

. 25/-. 10 

. 23/-. 03 

. 21/-. 05 

.  20/-. 07 

Trop.  N.  ATL 

. 24/-. 13 

. 26/-. 03 

. 29/+. 01 

.  28/-. 11 

Trop.  W.  PAC 

. 28/-. 04 

. 22/+. 01 

. 15/-. 04 

.  18/-. 03 

Trop.  Ind.  Ocean 

. 25/-. 32 

. 19/-. 40 

. 15/-. 46 

.  15/-. 45 

Tropics  (all) 

. 22/-. 08 

. 21/-. 08 

. 23/-. 39 

. 20/-. 38 

Trop.  N.  ATL 

. 23/-. 21 

. 21/-. 19 

. 13/-. 21 

.  12/-. 22 

Trop.  W.  PAC 

. 22/+. 02 

. 20/+. 34 

. 19/+. 05 

.  22/-^.  04 

Trop.  Ind.  Ocean 

. 11/-. 19 

. 12/-. 23 

. 09/-. 11 

.  18/  +  . 01 

Tropics  (all) 

. 24/-. 09 

. 22/-. 06 

. 21/-. 07 

.  20/-. 07 

Trop.  N .  ATL 

. 24/-. 16 

. 24/-. 10 

. 21/-. 09 

. 22/-. 16 

Trop.  W.  PAC 

. 25/-. 01 

. 21/+. 03 

.  17/'  +  .  30 

.  20  — . 30 

Trop.  Ind.  Ocean 

. 18/-. 26 

. 15/-. 32 

. 13/-. 33 

.  15/-. 29 

Error  reduction  scores  are  generally  highest  in  the  tropical 
Atlantic  and  definitely  lowest  in  the  Indian  Ocean.  The  lack  of 
great  variation  overall  between  T96  and  T168  is  noted  but  cannot 
be  explained.  The  generally  negative  bias  reflects 
underf orecasti.ng  of  25  kt  wind  speed  changes  at  230  mb,  the 
principal  exception  being  May  in  tropical  W.  PAC. 

2.1.2  Derived  Field  Skills. 

2. 1.2.1  1000  mb  Derived  Fields. 

2. 1.2. 1.1  Temperature  Ranges.  In  this  verification  the  ability 
of  the  model  to  forecast  the  probable  range  (upper-lower  limit' 
of  temperature  at  T120  was  accessed.  Ranges  were  obtained  in  two 


a.  3y  extracting  tne  ma x i m urn  ana  minimum  va.ues  at  eac.n 

grid  point  for  TAUs  96,  120  and  144  in  the  same  forecast 

run  to  obtain  a  T120  Time  Range . 

b.  By  extracting  the  maximum  and  minimum  values  from  the 

set  of  grid  point  values  for  TAUs  163,  144  and  123 

respectively  in  three  success i ve  00  GMT  forecast  runs  to 
obtain  a  T120  Lagged  Range . 

If  the  analyzed  value  ( not  the  range)  at  a  grid  point  was  within 
the  forecast  range,  a  "hit"  or  good  forecast  was  recorded.  Then 
the  percentage  of  grid  point  hits  was  calculated  for  the  two 
hemispheres.  The  results  for  1000  mb  temperatures  for  the  two 
months  combined  are  presented  below. 


RANGE 

PERCENT 

TYPE 

HEMISPHERE 

CORRECT 

Time 

Northern 

31 

Southern 

34 

Lagged 

Northern 

36 

Southern 

41 

Surprisingly,  more  skill  was  shown  in  the  S.  HEMI  than  in 
the  N.  HEMI.  Basic  field  RMS  errors  at  T120  correlated  with  the 
range  scores  above  in  April  and  for  both  months,  but  not  in  May 
when  RMS  errors  are  higher  in  the  S.  HEMI.  Basic  field  AC  scores 
at  T120  correspond  to  the  range  scores  in  April,  but  not  in  May 
or  for  both  months  when  the  AC  scores  are  higher  in  the  N.  HEMI. 
Interestingly,  the  lagged  ranges  are  consistently  better  than  the 
time  ranges  by  about  6  percentage  points,  but  36-41  percent 
correct  forecast  does  not  indicate  a  large  degree  of  skill. 


2. 1.2. 1.2  Wind  Direction  Ranges.  Wind  direction  ranges  were 
calculated  and  verified  as  discussed  under  Temperature  Ranges  for 


a 

\ 


all  five  major 

areas 

,  with 

the  following  result 

months  combined. 

RANGE  TYPE 

AREA 

PERCENT  CORRECT 

T  ime 

N. 

HEMI 

38 

N. 

PAC 

37 

N. 

ATL 

38 

S  . 

HEMI 

41 

TROP 

21 

Lagged 

N. 

HEMI 

39 

N. 

PAC 

37 

N. 

ATL 

40 

S. 

HEMI 

42 

TROP 

21 

The  scores  correlate  well  with  the  T120  RMS  scores  from  the 
1000  mb  basic  field  verifications.  In  this  category  the  lagged 
ranges  are  only  about  one  percentage  point  better  on  average  than 
the  time  ranges. 


2. 1.2. 1.3  Wind  Speed  Ranges.  Wind  speed  ranges  were  calculated 
and  verified  as  previously  discussed  for  ail  five  major  areas, 
with  the  following  results  for  the  two  months  combined. 


RANGE  TYPE 


Time 


P 


» 


Lagged 


AREA 
N.  HEMI 
N.  PAC 
N.  ATL 
S .  HEMI 
TROP 
N .  HEMI 
N.  PAC 
N .  ATL 
S.  HEMI 


PERCENT  CORRECT 

42 

43 
43 
42 
36 
47 
47 
47 
47 
38 


TROP 


scores  from  the  1333  mb  basic  field  verification  except  in  the 
Tropics  where  the  generally  lower  wind  speeds  account  for  the 
lower  RMS  errors.  In  this  category  the  lagged  ranges  are  4 
percentage  points  better  on  average  than  the  time  ranges. 

2. 1.2. 2  500  Millibar  Derived  Fields. 

2. 1.2. 2.1  Temperature  Ranges.  The  temperature  ranges  at  533  mb 
were  calculated  and  verified  as  they  were  at  1333  mb  {see 
subsection  2. 1.2. 1.1).  The  results  for  533  mb  temperatures  are 
contained  in  Taole  3. 

TABLE  3.  Temperature  Range  Scores. 


RANGE 

PERCENT 

PERIOD 

TYPE 

HEMISPHERE 

CORRECT 

April 

Time 

Northern 

33 

Southern 

29 

Lagged 

Northern 

36 

Southern 

32 

May 

Time 

Northern 

33 

Southern 

29 

Lagged 

Northern 

39 

Southern 

43 

Tota  1 

Time 

Northern 

32 

Southern 

29 

Lagged 

Northern 

38 

Southern 
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2. 1.2. 2. 2  Wind  Direction  Ranges .  Wind  direction  ranges 
calculated  and  verified  as  previously  discussed  for  all 
major  areas  and  the  results  are  presented  below. 


RANGE  TYPE 
Time 


AREA 


PERCENT  CORRECT 


Lagged 


N.  HEM I 

52 

N.  PAC 

51 

N.  ATL 

52 

S.  HEMI 

53 

TROP 

27 

N.  HEMI 

55 

N.  PAC 

54 

N.  ATL 

55 

S.  HEMI 

55 

TROP 

31 

The  scores  correlate  well  with  the  T120  RMS  scores  from  the 
1000  mb  standard  field  verifications  where  the  tropical  RMS 
errors  were  20  degrees  greater  than  in  the  other  major  areas.  In 
this  category  the  lagged  ranges  are  3  percentage  points  better  on 
average  than  the  time  ranges. 

2. 1.2. 2. 3  Wind  Speed  Ranges.  Wind  speed  ranges  at  500  mb  were 
calculated  and  verified  as  previously  explained,  with  the 
following  results  for  the  two  months  combined. 


RANGE  TYPE 


AREA 


PERCENT  CORRECT 


Again,  scores  tend  to  be  lower  in  the  tropics.  In  this 
category  the  lagged  ranges  verified  higher  than  the  time  ranges 
by  nearly  7  percentage  points. 

2. 1.2. 2. 4  Height  Truncations.  Spectral  representations  of  the 
533  mb  height  fields  were  prepared  for  both  analyses  ar.c 
forecasts  and  the  wave  number  (WN)  1  through  3  and  WN  4  through  9 
fields  were  compared.  This  was  done  for  both  hemispheres  and  for 
both  ocean  basins. 

RMS  errors  in  each  spectral  band  generally  range  from  about 
53  m  at  T96  to  near  73  m  at  T163  in  the  Northern  Hemisphere,  are 
somewhat  higher  in  the  Southern  Hemisphere,  and  are  more  often 
lower  in  magnitude  at  the  higher  wave  numbers.  However,  the 
expected  superiority  in  the  lower  (1-3)  wave  number  forecasts 

clearly  surfaces  when  one  considers  AC  scores.  WN  1-3  AC  scores 

average  3.12  higher  than  those  for  WN  4-9,  except  during  May  when 
the  higher  WN  forecasts  in  the  S.  HEMI  had  a  higher  AC. 

2. 1.2. 2. 5  Height  Averages.  In  this  verification,  averages  of 
multiple  height  forecasts  were  constructed  and  compared  to 
observed  values  of  530  mb  surface  height.  These  averages  were 
calculated  in  two  ways: 

a.  By  averaging  the  individual  grid  point  values  for  TAUs 

96,  120  and  144  in  the  same  forecast  run  to  obtain  a 

T123  Time  Average . 

o.  By  averaging  the  individual  grid  ooint  values  for  TACs 
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AC  SCORE  (  X  133) 


WAVE 

TIME 

LAGGED 

MONTH 

NUMBER 
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UNAVERAGED 

AVERAGE 

AVERAGE 
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1-3 

N.  HEMI 

64 

64 

68 

N.  PAC 

71 

71 

74 

N.  ATL 

61 

62 

68 

S.  HEMI 

45 

41 

52 

4-9 

N .  HEMI 

57 

57 

61 

N .  PAC 

57 

56 

59 

N.  ATL 

61 

59 

62 

S.  HEMI 

51 

50 

58 

ALL 

N.  HEMI 

66 

62 

66 

N.  PAC 

63 

61 

66 

N.  ATL 

72 

68 

73 

S.  HEMI 

54 

44 

53 

Average 

58.2 

57.9 

60.8 

There  appears 

little  to 

choose  between 

the 

unaveraged 

forecasts  and  the  time  averages. 

The  lagged  averages 

are  clearly 

superior  to  the  other 

averaged  score  in  the 

two,  with 

S.  HEMI  for 

the  exception  of 

April. 

the 

low  lagged 

2.2  Special  Verifications. 

2.2.1  Mean  Storm  Track. 

2. 2. 1.1  Evaluation  Procedure.  The  .NOGAPS  sea  level  pressure 

fields  for  TAUs  72,  96,  120,  144  and  168  were  used  to  create  a 

five-day,  centered  on  day  five,  composite  storm  track  field  over 
the  greater  North  Pacific.  The  area  is  the  same  26  x  31  subset 
of  the  standard  FNOC  63  x  63  Northern  Hemisohere  polar 


stereograpnic  grid  usee  oy  tne  ror  vortex  tracking  in  me 
Pacific  (see  Figure  2-32).  The  five  verifying  analysis  fields 
were  also  composited  and  the  resultant  storm  track  analysis  field 
was  used  to  compute  a  correlation  coefficient  between  forecast 
and  verifying  fields. 

Composites  were  constructed  by  removing  the  zonal  mean  from 
each  grid  point  in  each  forecast  sea  level  pressure  field.  Next, 
a  value  of  one  was  assigned  to  any  grid  point  whose  initial  value 
exceeded  the  zonal  mean.  Finally,  the  lowest  of  the  five  time 
values  at  each  grid  point  was  assigned  to  the  corresponding 
composite  grid  point.  Figure  2-03  shows  the  forecast  composite 
and  verifying  composite  for  a  typical  five-day  mean  storm  track 
centered  on  17  April  1984  (i.e.,  using  TAU  72,  96,  ...  168 
forecast  fields  prepared  from  00  GMT  data  on  12  April  and  00  GMT 
analysis  fields  from  15  through  19  April). 

The  Evaluation  Plan  stated  that  composites  would  also  be 
constructed  from  the  SEIS  vortex  tracking  program's  lows-only 
output.  This  was  not  done  because  substantial  changes  to  the 
vortex  tracking  program  were  in  progress  and  the  required  NEPRF 
personnel  were  not  available. 

2 . 2 . 1 . 2  Results .  Fifty-seven  five-day  composite  storm  track 
forecast  fields  were  verified.  The  first  was  prepared  on  28 
March  (for  the  five  day  period  31  March  through  4  April)  and  the 
last  was  prepared  on  8  June  (for  the  period  11-15  June).  The 
highest  correlation  coefficient  (0.84)  was  for  the  composite 
forecast  prepared  on  30  May  (see  Figure  2-04).  The  lowest  (0.44) 
was  for  the  21  May  forecast  (see  Figure  2-05).  The  mean 
correlation  coefficient  was  0.66  with  a  standard  deviation  of 
0.08.  Figure  2-06  is  a  plot  .of  all  57  correlations. 

The  considerable  variation  in  scores  indicates  the  ability 
of  this  evaluation  method  to  discriminate  between  "good"  and 
"bad"  forecasts  of  mean  storm  track.  One  also  notes  eoisodes  of 
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FIGURE  2-02,  The  63x63  Northern  Hemisphere  Polar  Stereograph 
and  the  26x31  Subset  used  for  Storm  Track  Verif 
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2.2.2  Area  Wind  Warnings. 

2. 2. 2.1  Evaluation  Procedure.  The  NOGAPS  five-  and  seven-day 
1300  mb  wind  fields  (u  and  v)  were  used  to  determine  if  gale 
force  winds  (greater  than  32  knots)  were  likely  over  any  of  eight 
7.5  degree  latitude  by  15  degree  longitude  ocean  areas  (four  in 
the  North  Pacific  and  four  in  the  North  Atlantic).  The  area 
locations  and  the  gale  yes-or-no  selection  and  scoring  criteria 
as  specified  in  subsection  2. 1.2. 2  of  the  Evaluation  Plan  were  as 
f o 1 lows : 

•  Location  of  areas : 

52. 5- 60. 0N/15. 0-30. 0W  and  175E-170W 

42. 5- 50. 0N/35. 0-50. 0W  and  155-170E 

32. 5- 40. 0N/60. 0-75. 0W  and  140-155E 

22. 5- 30. 0N/15. 0-30. 0W  and  150-165W 

(Eight  areas,  all  elongated  in  the  east-west  direction 
and  favoring  the  preferred  storm  tracks.) 

•  Forecast  Var iables : 

Gale  force  or  stronger  winds  in  area  on  day  5  -  yes 
or  no 

Gale  force  or  stronger  winds  in  area  on  day  7  -  yes 
or  no 

•  Selection  Criteria: 

Yes  if  10  or  more  of  the  28  grid  points  within  or  on 
the  perimeter  of  the  area  are  forecast  to  have  winds 
in  excess  of  32  knots  at  00  GMT. 

No  if  less  than  10  grid  points  meet  the  above 
criterion. 


If  gale  forecast  was  yes: 
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(The  30  and  36  knot  yes/no  verification  criteria  are 
purposely  in  the  "model's"  favor.) 

This  area  wind  warning  evaluation  commenced  on  25  March  and 
continued,  subject  to  forecast  and  verifying  field  availability, 
through  13  June  1984.  Fifty-three  five  day  and  51  seven  day 
NOGAPS  forecast  fields  were  verified  for  each  of  the  eight  areas. 

2. 2. 2. 2  Results.  Individual  area  contingency  tables  are 
presented  in  relative  plan  view  for  each  ocean  in  Figures  2-3~ 
and  2-08.  Ocean  basin  and  all-area  summaries  are  provided  in 
Figure  2-09.  The  latter  figure  also  shows  three  contingency 
tables  at  the  bottom  which  derive  from  a  more  straight-forward  33 
and  33  k t  scoring  criteria  than  the  30  and  36  kt  criteria  as  in 
subsection  2. 2. 2.1.  The  three  tables  at  the  bottom  of  Figure  2- 
39  consider  only  the  forecast  and/or  observation  of  13  or  more 
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FIGURE  2-09.  Wind  Warning  Summary  Scores  (30  and  36  knot  criteria 
at  top,  33  and  33  knot  criteria  at  bottom,-  five  d ac¬ 
counts  first,  seven  day  counts  second) 
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Of  the  12  "observed  aales"  (13  or  more  of  the  23  arid  ooints 


*  .  i  ..itt 


within  the  warning  area  with  more  than  32  knots  in  the  verifying 
analysis)  13  occurred  in  April.  All  12  gales  occurred 
four  higher  latitude  areas.  Thus,  about  half  of  the  f- 
and  half  of  the  areas  never  really  came  into  play. 


_  IS  o  _  o 


Less  than  half  of  the  actual  gales  were  forecast  ; 
lioeral  33-36  kt  criteria  (4  of  9  at  day  five  and  only  2  o: 
day  seven)  and  far  fewer  by  the  33-33  kt  criteria  (only  3 
at  day  five  and  none  at  day  seven). 


The  all-area  false-alarm  rate  varied  from  a  low  of  32%  (day 


ve ,  33-36  kt  criteria)  to  100%  (day  seven,  33-33  kt  criteria). 


Taola  5  provides  threat  and  Heidke  skill  scores 
sets  of  summary  data. 
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TABLE  5.  Gale  Forecasting  Skill  Summary. 
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Gale  F or ecast  i  n 


FIVE  DAY  FORECASTS 

All  Areas  (30-36  kt 

cr i ter i a ) 

N.  Pacific  Areas  (33-33  kt 

criteria) 

N.  Atlantic  Areas  (33-33  kt 

criteria) 

All  Areas  (33-33  kt 

criteria) 

SEVEN  DAY  FORECASTS 

N.  Pacific  Areas  (30-36  kt 

criteria) 

N.  Atlantic  Areas  (30-36  kt 

criteria) 

All  Areas  (30-36  kt 

criteria) 

N.  Pacific  Areas  (33-33  kt 

criteria) 

N.  Atlantic  Areas  (33-33  kt 

criteria) 

All  Areas  (33-33  kt 

criteria) 


AD JEST EC 
THREAT  THREAT 

SCORE  SCORE1 


0.148 

0.  130 

0.235 
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-0.044 
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0.  133 
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0.204 

0.  100 

0.074 
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-0. 015 

-0. 022 

0.083 

-0. 062 
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-0.025 

-0.033 

0.000 

-0. 036 

-0.040 

0. 000 

-0. 030 

-0. 037 
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threat  score 
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TS 

1 


Threat  Score  (ATS)  is  derived  from  the  standard 
(TS )  as  f ol lows  : 

-  (observed  occurrences/possible  occurrences) 

-  (observed  occurrences/possible  occurrences) 


By  all  of  the  above  measures,  the  model's  ability  t: 
forecast  gales  during  the  evaluation  was  marginal  for  day  five 
and  essentially  non-existent  for  day  seven.  Somewhat  more 


encouraging  resu.os  are  octal r. 
onlv  the  hicher  latitude  areas. 


w n e n  one  considers 


cor  example,  t : 


two  most  northerly  Pacific  areas  (?1  and  ?2 )  combined  using  33-35 
k t  criteria  are: 


THREAT 

ADJUSTED 

HEIDKE 

SCORE 

THREAT  SCORE 

SKILL  SCORE 

Five  Day  Forecasts: 

0.25 

0.22 

0.  37 

Seven  Day  Forecasts: 

0.20 

0.  15 

0.29 

In  addition  to  the  high  false-aiarm  rates  cited  earlier,  the 
larger  number  of  over  forecasts  (forecast  yes  -  observed  no)  than 
under  'forecasts  (forecast  no  -  observed  yes)  and  the  generally 
lower  over  forecast  percentage  {Avg  %)  scores  both  indicate  a 
decided  model  tendency  to  generate  overly  strong  winds  at  1000  mb 
in  middle  latitudes.  All  but  one  of  the  false  alarms  occurred  in 
March  or  April.  In  April  the  mean  1000  mb  wind  speed  errors  at 
TAUs  120  and  163  were  2.1  and  1.3  kt  respectively.  Corresponding 
scores  for  the  Atlantic  were  +3.9  and  +0.6. 

2.2.3  Subjective  Sensible  Weather  Forecasts. 

2. 2. 3.1  Evaluation  Procedure.  Selected  NOGAPS  analysis  and 
forecast  fields  (those  identified  with  P  notation  in  Table  2-01 
of  the  Evaluation  Plan)  were  plotted,  when  available,  and 
displayed,  together  with  recent  synoptic  reports,  each  Monday 
through  Friday.  They  were  used  as  guidance  to  prepare  five-  and 
seven-day  forecasts  of  several  operationally  significant  sensible 
weather  parameters.  The  parameters  and  other  details  concerning 
this  part  of  the  evaluation  are  discussed  in  subsection  2.  1.2. 3 
of  the  Evaluation  Plan. 

The  nine  forecast  locations  chosen  are  listed  in  the 
following  table: 
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AREA 

LOCATION 

3 LOCKS TAT  ION 

N E  Pacific 

Kodiak,  Alaska 

7  3  3  5  0 

Adak,  Alaska 

734  54 

Astoria,  Oregon 

72791 

NW  Atlantic 

Sable  Island,  Canada 

7 1 6  0  0 

Key  West,  Florida 

72201 

Hatteras,  No.  Carolina 

72304 

N E  Atlantic 

Valencia,  Ireland 

03953 

Keflavik,  Iceland 

04018 

La  Coruna,  Spain 

08001 

A  total  of  16  five-  and  16  seven-day  forecasts  were  prepared 
for  each  location  -  the  first  on  12  April  and  the  last  on  8  June. 
It  was  planned  that  sensible  weather  forecasts  would  be  prepared 
on  Tuesdays  and  Thursdays,  thus  permitting  comparisons  of  each 
Tuesday  seven-day  forecast  with  a  Thursday  five-day  forecast 
verifying  at  the  same  time.  Unfortunately,  various  problems 
involving  the  model  and  computer  availability  usually  caused  one 
day  or  the  other  to  be  missed  and  only  3  of  3  potential  Tuesday- 
Thursday  pairs  were  realized. 

The  questionnaire  shown  in  Figure  2-10  was  used  to  obtain 
pre-evaluation  period  information  from  individual  forecasters. 
All  of  the  participants  were  experienced  Navy  aviation  and  marine 
weather  forecasters,  but  none  were  currently  or  recently  active. 
Individual  experience  levels  ranged  from  eight  to  17  years.  All 
but  one  of  the  four  had  a  degree  in  meteorology  (one  BS,  one  MS 
and  one  PhD)  and  the  non-degree  forecaster  had  completed 
graduate  courses  in  meteorology.  Forecaster  familiarity  with 
assigned  areas  and  stations  varied  from  nil  in  one  case,  through 
"very  familiar"  with  one  of  three  assigned  stations  and  "somewhat 
familiar"  with  the  others,  to  very  familiar  with  the  entire  area. 
One  of  the  four  forecasters  was  very  familiar  with  the  NOGAPS 
model  output  prior  to  the  evaluation  and  had  been  working  with  it 
on  a  nearly  daily  basis  for  many  months.  Two  of  the  other  three 
forecasters  were  totally  unfamiliar  with  NOGAPS  output  prior  to 
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HEPIUHHttMSe  FORECAST  EVALUATION  <MRFE)  FORECASTER  QUEST  I  CM**!  RE 


NAME :  - 


DATE  : 


MOST  SIGNIFICANT  EDUCATION  AND/OR  TRAINING  IN  ATMOSPHERIC 
SCIENCE:  _ 


YEARS  AND  TYPE  OF  WEATHER  FORECASTING  EXPERIENCE: 


AREA  ASSIGNED  FOR  MRFE :  _ 

BLOCK  STATIONS  AND/OR  SUBAREA <  S)  SELECTED: 


TO  WHAT  DEGREE  WERE  YOU  FAMILIAR  WITH  YOUR  AREA  AND  STATIONS 
PRIOR  TO  THE  MRFE?  _ 


TO  WHAT  DEGREE  WERE  YOU  FAMILIAR  WITH  THE  NOGAPS  MODEL  OUTPUT 
PRIOR  TO  THE  MRFE?  _ 


First  pace  of  the  Medium-Range  Forecast  Evaluation 
(MRFE)  Forecaster  Questionnaire. 


FIGURE  2-10. 


the  evaluation.  The  rourt.n  was  ra.tuuar  witn  tne  model,  out  nao 
never  used  the  output  to  forecast. 

2. 2. 3. 2  Results.  Verifications  of  the  sensible  weather 
forecasts,  including  ITCZ  forecasts,  are  discussed  below  on  a 
variable- by -variable  basis.  With  two  minor  exceptions,  all  non- 
I TCZ  scoring  was  as  specified  in  Table  2-03  of  the  Evaluation 
Plan  (reproduced  here  as  Figure  2-11).  The  exceptions  (Rain  or 
Shower  and  Frozen  Precipitation)  are  noted  within  the  parameter 
results  discussion. 

2. 2.3. 2.1  Winds  >25  kt  -  yes/no.  The  results  of  the 
verification  are  presented  in  Table  6.  (Y/Y  is  greater  than  25 
kt  forecast  (yes)  and  Verified  (yes),  Y/N  is  greater  than  25  kt 
forecast  (yes)  but  not  verified  (no),  N/N  is  not  forecast  and  not 
verified,  etc.) 

TABLE  6.  Skill  in  Forecasting  Surface  Wind  >25  kt. 


FORECAST/VERIFICATION  COUNTS 
FIVE-DAY  FCSTS  SEVEN-DAY  FCSTS  ALL  FORECASTS 


LOCATION 

Y/Y 

N/N 

Y/N 

N/Y 

Y/Y 

N/N 

Y/N 

N/Y 

Y/Y 

N/N 

Y/N 

N/Y 

Kodiak 

— 

16 

—  — 

— 

— 

15 

1 

— 

— 

31 

1 

— 

Adak 

— 

14 

2 

— 

1 

14 

1 

— 

1 

28 

3 

— 

Astoria 

— 

16 

— 

— 

— 

16 

— 

— 

— 

32 

— 

— 

All  NE  PAC 

— 

46 

2 

— 

1 

45 

2 

— 

1 

91 

4 

— 

Sable  Island 

— 

15 

— 

1 

— 

14 

2 

— 

— 

29 

2 

1 

Key  West 

— 

16 

— 

— 

— 

16 

— 

— 

— 

32 

— 

— 

Hatteras 

— 

16 

— 

— 

— 

15 

1 

— 

— 

31 

1 

— 

All  NW  ATL 

— 

47 

— 

1 

— 

45 

3 

— 

— 

92 

3 

1 

Valencia 

— 

16 

— 

— 

— 

14 

2 

— 

— 

30 

2 

— 

Kef  la vi k 

— 

15 

1 

— 

— 

16 

— 

— 

— 

31 

1 

— 

La  Coruna 

— 

16 

— 

— 

— 

16 

— 

— 

— 

32 

— 

— 

All  NE  ATL 

— 

47 

1 

— 

— 

46 

2 

— 

— 

93 

3 

— 

TOTAL 

— 

140 

3 

1 

1 

136 

7 

_ 

1 

276 

10 

1 

T*T 


T 


PARAMETER 
Winds  >  25  .<s 

Avg  Sfc  Wind  Spd 
Avg  Sfc  Wind  Dir 

Sfc  Air  Temp 

Avg  Cloudiness 

Lowest  Cloud  Base 

Precip  Expected 

Rain  or  Shower 


Frozen 


Sfc  Visibility 


FIGURE  2 


SCCRIh'G  RULES 

If  fcst  yes:  any  13-062  report  >_  20  scores  1, 
otherwise  0 

If  fcst  no :  any  13-062  report  30  scores  0, 

otherwise  1 

Fcst  value  -  avg  of  13-062  reports  =  score  (with  sign) 


If  fcst  quadrant  =  geometric  avg  of  13-062  score  1, 


otherwise  0 

Fcst 

value 

-  avg  of  21Z-03Z  reports  (deg  F)  = 
score  (with  sign) 

Fcst 

value 

-  avg  of  18-06Z  total  reported  clouds 
(N)  =  score  (with  sign) 

Fcst 

value 

-  avg  of  1S-06Z  reported  bases  (h  as 

"plotted")  =  score  (with  sign) 

If  fcst  yes:  any  13-062  ww>49  or  any  21-062  W>4 
scores  1,  otherwise  0 

If  fcst  no;  more  than  one  18-062  ww>49  or  21-062  W>4 
scores  0 ,  otherwise  1 

If  precip  score  =  0  this  score  is  N’  (null) 

If  precip  score  =  1  and  precip  fcst  '-'as  no,  this 

score  is 

Otherwise-.  13-06Z  avg  of  ww>49  and  (VixlO)  >40  =  PTA 
Then : 

If  fcst  was  rain  and  PTA<7~  score  1 
If  fcst  was  shower  and  ?TA>73  score  1 
Otherwise  score  0 

If  precip  score  =  0  this  score  is  N  (null) 

If  precip  score  =  1  and  precip  fcst  was  no,  this 

score  is  M 

Otherwise  if  fcst  ves:  any  13-C62  ww  56-57,  66-~9, 

83-90  or  93-97  or  99  or  any  21-062  W  of  7 
scores  1,  otherwise  0 

if  fcst  no:  less  than  two  of  the  above 

scores  1,  otherwise  0 

If  fcst  was  yes  and  any  21-032  coded  W<56  (^4  mi) 
score  1 

If  fcst  was  no  and  all  21-03Z  coded  W>35  (ny-G1*  mi) 
score  1 

Otherwise  score  0 


11.  Sensible  Weather  Forecast  Scorinc  Criteria 
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As  in  the  area  wind  warning  verification  discussed  in 
subsection  2.2.2,  these  results  reflect  a  near  total  absence  of 
strong  winds  for  the  Northern  Hemisphere  middle  latitudes  during 
this  spring  evaluation.  Ten  false  alarms  (Y/N) ,  one  strong  wind 
event  correctly  forecast  (Y/Y)  and  one  event  missed  (N/Y)  are  not 
a  statistically  sufficient  sample  from  which  to  assess  skill, 
although  the  tendency  to  over- forecast  a  strong  wind  event  is 
apparent . 


2. 2. 3. 2. 2  Average  Surface 

Wind  Speed. 

The  results 

of  this 

verification  are 

presented 

in  Table  7. 

TABLE 

7.  Surface 

Wind  Speed 

Forecasting  Skill 

• 

MEANS/STANDARD 

DEVIATIONS  (KT) 

DAY 

FIVE 

DAY  SEVEN 

LOCATION 

OBSERVED 

ERROR 

OBSERVED 

ERROR 

Kodiak 

10.4/3.0 

-0.6/4 . 4 

10.2/4.7 

-1. 1/5.9 

Adak 

9.3/4. 1 

+  1. 6/5.  2 

3. 2/4. 0 

+2. 9/5. 6 

Astoria 

8. 6/1. 5 

-0. 4/2. 4 

9. 7/1. 7 

-1.9/3. 8 

All  ME  PAC 

9. 5/3.  1 

+0. 6/4. 3 

9. 4/3. 3 

-0. 0/5. 6 

Sable  Island 

13 . 9/7. 2 

-0. 1/3.6 

10.4/2.7 

1-  2  .  5  /  5  .  5 

Key  West 

9. 3/1. 5 

-2. 6/3. 9 

8. 4/3. 6 

-2.3/4. 2 

Hatteras 

3. 9/2.  1 

+1 . 5/4 . 7 

9. 7/3.  1 

+1. 3/7. 0 

All  NW  ATL 

10.7/5.0 

-1. 1/6. 2 

9. 5/3. 3 

+0. 5/6. 0 

'/a  lenc  i  a 

5. 8/3 . 9 

+4 .7/4.7 

9. 5/5. 4 

+2. 1/8. 4 

Kef lavik 

9. 1/4. 4 

+2. 8/7. 0 

8. 6/4 . 1 

+3. 3/5. 4 

La  Coruna 

5. 6/4. 2 

-0. 6/4 . 0 

4. 3/4 . 2 

+0. 0/6. 5 

All  NE  ATL 

7.0/4  .  5 

+  2. 4/5. 7 

7. 5/5. 0 

+1.3/7. 0 

TOTAL 


9. 0/4 . 5 


+•0  .  5/5 . 6 


8.  3/4 . 2 


+0.8/6. 3 


From  the  observed  averages  and  standard  deviations  (3D)  one 
sees  that  winds  were  generally  low  -  between  5  and  15  knots, 
except  for  Sable  Island  where  they  were  a  few  knots  higher  and  La 
Coruna  where  they  were  lower.  Average  errors  were  small  but  not 
indicative  of  any  particular  skill  when  one  considers  the 
standard  deviation  (SD)  of  the  error.  In  all  but  one  case  (La 

Coruna  -  Day  Five)  the  error  SD  is  greater  than  the  observed  SD 
which  may  mean  that  always  forecasting  the  "average"  wind  speed 
would  have  shown  more  skill.  To  satisfy  the  acceptable  levels  of 
accuracy  specified  in  Table  1-01  of  the  evaluation  plan,  the 
error  SD  should  be  about  25  to  35%  of  the  observed  3D. 

2. 2. 3. 2. 3  Average  Surface  Wind  Direction  (quadrant).  The 

results  of  this  verification  are  presented  in  Table  8. 

TABLE  8.  Surface  Wind  Direction  Forecasting  Skill. 

COUNTS 

DAY  FIVE  DAY  SEVEN  ALL  FORECASTS 


LOCATION 

HIT 

MISS 

NULL 

HIT 

MISS 

NULL 

HIT 

MISS 

NULL 

Kodiak 

4 

12 

— 

2 

14 

— 

6 

26 

-- 

Adak 

5 

11 

— 

4 

12 

— 

9 

23 

— 

Astoria 

4 

12 

— 

4 

12 

— 

8 

24 

-- 

All  NE  PAC 

13 

35 

-- 

10 

38 

— 

23 

73 

— 

Sable  Island 

3 

13 

— 

2 

14 

— 

5 

27 

— 

Key  West 

5 

11 

— 

4 

11 

1 

9 

22 

1 

Hatteras 

5 

10 

1 

3 

13 

-- 

8 

23 

1 

All  NW  ATL 

13 

34 

1 

9 

38 

1 

22 

72 

2 

Valencia 

3 

13 

— 

6 

9 

1 

9 

22 

1 

Kef  lav  lk 

6 

10 

— 

6 

10 

— 

12 

20 

-- 

La  Coruna 

7 

6 

3 

5 

2 

9 

12 

8 

12 

All  NE  ATL 

16 

29 

3 

17 

21 

10 

33 

50 

13 

TOTAL 

42 

98 

4 

36 

97 

11 

78 

195 

15 

43 


Ignoring  the  nu_ is  which  resulted  iron  cal.t  wind  forecasts,  on _ y 
29  percent  (73/273)  of  the  forecasts  were  correct.  The  five-day 
forecasts  were  only  .marginally  better  than  the  seven-day  (32% 
versus  27%).  Since  any  observed  average  direction  within  43 
degrees  of  the  forecast  average  direction  was  considered  correct, 
these  results  are  far  from  encouraging.  In  only  one  instance  (La 
Coruna)  did  the  number  of  correct  forecasts  exceed  the  number 


incorrect . 

These 

results  do  not  satisfy  the  day-five 

level  of 

accuracy  specified 

in  Table  1 

-01  of  the 

evaluation  plan. 

2 . 2 . 3 . 2 . 4  Surface 

Air  Temperature. 

The  results 

of  this 

verification 

are  presented  in 

Table  9 . 

TABLE 

9 .  S  ur f ace  Air 

Temperature 

Forecasting  Skill. 

MEANS/STANDARD 

DEVIATIONS  (Deg 

.  F) 

DAE 

FIVE 

DAY 

SEVEN 

LOCATION 

OBSERVED 

ERROR 

OBSERVED 

ERROR 

Kodiak 

46.1/8. 5 

-1. 6/5. 8 

47. 3/5.6 

-1. 4/5. 7 

Adak 

42.9/3.3 

-3. 2/3. 4 

45.2/4.7 

-4. 0/4. 0 

Astoria 

54.6/4.7 

-3.4/5. 5 

53 . 6/4 . 0 

-1 . 6/4 . 3 

All  NE  PAC 

48.1/7.8 

-2. 7/5. 1 

48.9/6.0 

-2.7/4.  8 

Sable  Island 

45. 1/5. 2 

+3 .8/8.9 

45.7/5.0 

+  2 . 1/9.  2 

Key  West 

78.4/2.3 

-0. 1/3.6 

78.5/2.3 

+0.1/3. 5 

Hatteras 

67.7/6. 3 

-1.9/7. 5 

67 .3/6.7 

-2 . 6/9. 2 

All  NW  ATL 

63.2/14.9 

-1. 3/6. 5 

63.3/14.7 

-1. 7/7. 3 

Valencia 

48 . 8/5.5 

+1. 3/3. 2 

51.2/4.8 

-0 . 6/4 . 9 

Ke  f 1 a v  i  k 

40.9/5.2 

-0. 6/4. 2 

41.1/4.6 

+  0.9/4.  5 

La  Coruna 

53 .4/4.0 

-0.9/3. 9 

54 .4/4.3 

+0 . 3/5. 2 

All  NE  ATL 

47.7/9.0 

-0. 1/3. 9 

48 . 9/7. 3 

-0.0/4 . 9 

TOTAL 


51 . 5/9.3 


-3. 7/5.  3 


53 . 3/1 1 . 3 


-J.  9  '6  .  2 


^  o  m  oa  r  i  n  c  5"3nca  rci  ceviaci  3ns  c^.1  ,  :ne  5905  1  r.  a  -  :r.e  errrr 
SD  are  higher  than  for  the  observations  at  4  of  3  locations  on 
Day  Five  and  at  7  of  9  at  Day  Seven.  Assuming  that  the  SD 
observed  over  the  eight  weeks  of  forecasting  fairly  represents 
the  Seasonal  Standard  Deviation  specified  in  Table  1-J1  of  the 
Evaluation  Plan,  it  is  clear  that  the  acceptability  criteria  of 


0.4  at  Day 

Five  and  approximately  0.6  at 

Day  Seven 

were  not 

achieved. 

The  best  at  Day 

Five  was  0.53 

for  Valencia 

{3.275.5) 

and  the  best 

at  Day  Seven  was 

0.35  for  Adak 

• 

2. 2. 3. 2. 5  Average  Cloudiness 

(eighths) . 

The  results 

of  this 

verification 

are  presented  in 

Table  10 . 

TABLE  10.  Average  Cloudiness  Forecasting  Skill 

• 

MEANS/STANDARD 

DEVIATIONS  (3 

THS  ) 

DAY 

FIVE 

DAY 

SEVEN 

LOCATION 

OBSERVED 

ERROR 

OBSERVED 

ERROR 

Kod iak 

6. 3/2. 2 

+3. 4/2. 3 

6. 3/1. 6 

+  1.1/2.  0 

Adak 

7. 3/0.7 

-1. 0/2. 3 

7. 6/0. 8 

-2. 1/2.7 

Astoria 

5. 9/1. 3 

-0. 3/2. 8 

6. 9/1.1 

-0.4/3. 2 

All  NE  PAC 

6. 5/1. 8 

-0. 2/2.6 

6. 9/1. 3 

-0. 1/2.6 

Sable  Island 

5. 7/2. 6 

-0. 3/3. 5 

5. 3/2. 6 

+  3. 4/3.  6 

Key  West 

4. 3/2. 3 

-2.6/2. 8 

3. 9/2. 2 

- 1 .  4  /  3  .  1 

Ha  t  te  r as 

3. 5/3. 2 

+1. 2/3. 5 

4. 3/3.0 

-1. 6/3. 3 

All  NW  ATL 

4. 5/2. 9 

-0. 6/3.7 

4. 5/2. 7 

-1 . 1/3 . 6 

Valencia 

4. 9/2. 7 

+0. 6/2.5 

5.9/2. 6 

-0. 5/3.  9 

Kef lavik 

6.9/1. 5 

-0.7/1. 9 

6. 6/1. 8 

-0 . 4/2 . 8 

La  Coruna 

5. 6/2. 8 

-2. 4/3. 8 

4.8/3. 1 

-0. 6/4. 6 

All  NE  ATL 

5. 8/2. 5 

-0.8/3. 1 

5. 7/2. 7 

-0. 5/3. 3 

TOTAL 

5. 6/2. 6 

-0. 5/3. 2 

5  .  8  /  2 .  5 

-0.7  '3.4 

assuming  a  r.or^a.  t  .  s  t  r  :  u  _  t  .  ,  n  __  5;^.  t*  ,  -..-ruo  .. 
the  seven-day  Kodiak  f  trecasts  and  m ;  re  than  two  or. :  r  is  :  f  me 
five-day  Xeflavik  forecasts  .vet  the  _y2  3  accuracy  criteria  set 
forth  in  Table  1-31  of  tne  Evaluation  Plan.  The  ct.oer  .6 
opportunities  were  .missed.  However,  even  in  the  two  successes 
cited,  a  forecast  of  "average"  cloudiness  might  have  oeen  tetter 
since  the  standard  deviation  (3D)  of  the  observations  is  less 
than  the  SD  of  the  error.  The  latter  general  indication  of  no 
skill  is  evident  for  all  but  one  of  the  13  station /forecast- day 
pairs  (the  exception  being  Vaiencia/day  five). 


2. 2. 3. 2. 6  Lowest  Cloud  3ase. 
are  oresented  in  Table  11. 


he  resu.ts  of  tms 


TABLE  11.  Lowest  Cloud  Base  Forecasting  Skill 


MEANS/STANDARD  DEVIATIONS  (FT  x  130) 


DAY 

FIVE 

DAY 

SEVEN 

LOCATION 

OBSERVED 

ERROR 

OBSERVED 

E  R  R  0  R 

Kod i ak 

25.1/19.7 

-11.3/20.3 

28 . 9/17. 3 

-16.3/17.9 

Adak 

12.0/7.8 

+2.5/7. 1 

11.1/6.5 

+2.7/7. 1 

Astoria 

23.0/20. 3 

-7.7/20. 1 

19.9/16. 3 

-4 . 6/15. 5 

All  NE 

PAC 

20.1/17.9 

-5.5/17.9 

20.3/16.2 

-6.2/16.4 

Sable  Isl 

and 

25.3/27.4 

+12 . 3/44.4 

49.5/31.6 

-14 . 5/43 . 3 

Key  West 

23.4/19.0 

+56.2/22.1 

25.9/19.7 

+57.3/17.5 

Hatteras 

63 . 1/19.8 

-22.4/37. 4 

56 . 9/23 . 0 

-1.9/34.  1 

All  NW 

ATL 

37.5/28.8 

+12.7/43.2 

44.1/28. 5 

+13.7/45.7 

Va lenc  i  a 

28. 3/19.0 

+1.4/25. 4 

19.9/22. 3 

+3.2/30.2 

Kef lavik 

18.2/15.2 

+8.1/23.4 

18.6/18. 5 

+9.3/29.6 

La  Coruna 

27.6/25.9 

+12. 3/32.3 

30 . 3/27 . 4 

-3.0/37.0 

A  -  -  M  E 

ATL 

24.7/21.0 

+7.2/27.5 

22.9/23.3 

-3.3  33.0 

TOTAL 

27. 4/24 . 2 

4.9/34.3 

29. 3/25. 7 

3.3  34.0 

wn  ■  i  n  mi'inL  »U 


ty  m  u"  l  ■  -  '  l  ■>  1 


>  v  '-v  -  v-  •■  v  -  v  ■ 


:n 


-  *VtT5  - 


r .-o  _  o 


1  -»  ^  -  - 


s  olot:edM  value  fur 
ud  is  3  3  (3,333  fe 

signer  values  were  converted  to 
computing  the  absolute  error.  Thus,  the  maximum  possiole  error 
is  33  ft  :<  133.  In  this  verification,  as  in  others  a  cove ,  tost 
scores  would  have  been  improved  by  forecasting  the  observed  mean. 
In  only  4  of  the  13  st a t i on/ f or ecas t-day  pairs  was  the  standard 
deviation  of  the  error  (error  SD)  less  than  the  observed  S3,  and 
in  all  four  cases  the  difference  is  less  than  the  mean  error.  Nr 
skill  at  forecasting  cloud  base  is  evident. 

2. 2. 3. 2. 7  ? r e c i o  i  t a t  i  on  Exoected  -  ves/no.  Taole  12  summaric-s 


me  results  or  t.n: 


;  i  on  . 


TABLE  12,  Pr ec  i  o  i  ta t i on  Yes/Mo  Forecasting  Skill. 


FORECAST /VERIFICATION  COUNTS 


FIVE  DAY  FCSTS 


a 

lj  .  ..  ^ 

Y  7Y 

N/N 

Y/N 

N/Y 

Kod i ak 

6 

— 

— 

A 

-x 

5 

Adak 

3 

— 

— 

5 

7 

*,  - 

Astoria 

4 

2 

1 

6 

4 

•i 

All  NE  PAC 

18 

2 

1 

15 

16 

Sable  Island 

4 

2 

— 

6 

3 

Key  west 

— 

3 

— 

— 

— 

• 

Hatteras 

1 

1 

— 

2 

— 

All  NW  ATL 

5 

5 

— 

3 

3 

Valencia 

3 

1 

1 

2 

5 

• 

Ke  f 1  a  v  i  k 

13 

— 

— 

2 

13 

La  Coruna 

4 

1 

— 

6 

5 

All  NE  ATL 

22 

2 

i 

13 

2  3 

• 

TOTAL 

4  5 

10 

2 

33 

3  9 

SEVEN  DAY  FCSTS 
N/N  Y/N  N/Y 


ALL 


2 

j 

9  14 


o 

6 

13 

3 
2 
5 

13 

5 

2 

4 


11  -- 
15 
3 
34 


1 

3 

13 

23 

9 

42 

34 
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_ounts  ao  net  equa.  io  per  station  per  rorecast  ray  sees  use 
precipitation  data  were  missing  from,  many  synoptic  reports.  Here 
we  see  a  clear  tendency  to  under-forecast  the  event  i  5~  total 
N/Y's  versus  only  7  Y/N's).  This  was  true  for  all  stations  and 
areas  . 


Precipi tat i on 

threat  scores 

(TS)  and 

ad  j  us  ted 

4- 

hreat 

scores 

(ATS)  are  presented 

i  n 

Table  13. 

(Note , 

ATS  is 

de 

fined 

by  the 

footnote  to  Table  5 

.  ) 

TABLE  13. 

P  r  e 

cipitation 

Eorecas 

ting  5  k  i  1 

1 

5  umma  r 

V  . 

PRECIPITATION  E0RECA5 

TS 

5 

DAY 

7 

DAY 

BOTH 

DAYS 

LOCATION 

TS 

ATS 

TS 

ATS 

TS 

ATS 

Kodiak 

0.  60 

undef 

3.71 

undef 

3.65 

undef 

Adak 

0.62 

undef 

3  .  53 

-4.  46 

3.  63 

-9.43 

Astoria 

0.36 

-1.77 

3.43 

-2.63 

3  .  33 

-2.13 

All  ME  PAC 

0.  53 

-4.64 

0.  55 

-3.30 

3.  54 

-4.21 

Sable  Island 

0.43 

-2.63 

3.  33 

-3.46 

3.  37 

-1.07 

Key  West 

0.00 

3.00 

3.00 

-3.  50 

0.  00 

-3.  40 

Hatteras 

0.3  3 

-1 .  68 

3.00 

-2  .  50 

0  .  13 

-2  .  19 

All  NW  ATL 

0.  33 

-3.91 

0.  19 

-0.77 

0.  23 

-3.32 

Valencia 

0.73 

-0.62 

0.42 

-1.51 

0.57 

-1  .  15 

Ke  f la  v  i  k 

0.83 

undef 

3.  33 

undef 

0.83 

undef 

La  Coruna 

3.  40 

-5.  63 

0.  56 

-0.76 

0.47 

-2.  39 

All  NE  ATL 

0.67 

-2.85 

0.61 

-1.81 

0.64 

-2.19 

Kod i a  k 

3.00 

0. 33 

0. 30 

3. 00 

3.33 

Adak 

3.3  3 

3.  30 

3.17 

0.10 

3  .  13 

Astoria 

0.22 

-3 . 31 

3.  25 

3.  13 

3.24 

Ail  N  E  ?AC 

3.11 

3.33 

0.19 

3.13 

3.15 

Sable  Island 

0.2  5 

3  .  13 

3  .  25 

-3  .  3 7 

7  ‘7  “ 

Key  West 

I.  03 

undef 

3. 67 

3.3  3 

7  - 

Hatteras 

3.3  3 

3.  11 

3.29 

0  .  33 

3  .  o  J 

All  NW  ATL 

0.43 

0.  16 

0.38 

-0.  14 

3.43 

Valencia 

3.  25 

3.  10 

3.  13 

-3  .  13 

3  .  1~ 

Kef lavik 

3.00 

3.30 

3.  03 

3.  30 

3.33 

La  Coruna 

3.  14 

3.36 

3.  33 

3.19 

3.23 

Ail  K E  ATL 

3.  15 

3.07 

3.19 

3.  36 

~i  '  — 

TOTAL 

0.  22 

0.  10 

0.  26 

3.  37 

3.25 

Both  the 

precipitation 

threat 

scores 

a  nd , 

precipitation  was 

so  frequent,  t 

he  no  ore 

cioitati 

on  threa 

are  presented. 

Since 

ATS  =3 

approx  i  m 

ates  ra 

neon  c  h 

negative  ATS  indicates  less  skill  than  chance  and  AT3  =  1  w 
perfect;  the  precipitation  forecasts  can  be  seen  to  hav 
to  no  skill. 


2 , 2 . 3 , 2 . 3  Pain  or  Shower.  Table  14  is  the  result 
verification.  (R/R  is  rain  forecast  and  rain  verified, 


ram  for- 


and  shower  verified,  etc.! 


« 


I 


TAB 

LE  1 

4 .  Rain  or 

Showe r 

F  o  r  e  c  a  s 

ting 

Skill 

• 

FORE 

CAST, '7 

ERI FICAT 

ION 

COUNTS 

FIVE 

DAY  FCSTS 

SE7S 

N  DAY  FC 

STS 

ALL 

FORECAS 

TS 

LOCATION 

R/R 

S/S  R/S  S/R 

R/R 

S/S  R/S 

S/R 

R/R 

S/S  R/S 

S/R 

Kodiak 

1 

3  2 - 

1 

4 - 

— 

2 

7  2 

— 

Adak 

3 

4 -  1 

3 

1 - 

3 

6 

5 - 

4 

Astoria 

1 

2  1 - 

— 

3  1 

— 

1 

5  2 

— 

All  NE 

PAC 

5 

9  3  1 

4 

8  1 

3 

9 

17  4 

4 

Sable  Island 

1 

2 -  1 

1 

1 - 

1 

2 

3 - 

2 

Key  West 

— 

— 

- - 

— 

— 

- - 

— 

Hatteras 

— 

1  —  — - 

— 

- - 

— 

— 

1 - 

— 

All  MW 

ATL 

1 

3 -  1 

1 

1 - 

1 

2 

4 - 

2 

Valencia 

1 

3  3  1 

3 

1  --- 

1 

4 

4  3 

2 

Kef  lav  ik 

5 

-  2  3 

5 

3  1 

1 

10 

3  3 

4 

La  Coruna 

1 

2  1 - 

— 

3 - 

2 

1 

5  i 

2 

All  NE 

ATL 

7 

5  6  4 

8 

7  1 

4 

i  5 

12  7 

3 

TOTAL 

13 

17  9  6 

13 

16  2 

8 

26 

33  11 

14 

The 

counts 

in  this  verification  are  1 

ow  be 

cause  e 

ach 

requ ires 

both 

the  forecast 

and 

the  reporte 

d  occ 

u  r  rente 

-  P 

precipitation.  Also,  a  computed  "precipitation  average  amount" 
(PTA)  as  specified  in  the  Evaluation  Plan's  scoring  roles  proved 
unreliable.  Instead  a  count  of  reported  present  weather  showers 
(ww  greater  than  79)  and  reported  past  weather  showers  w  greater 
than  7)  was  made.  Half  or  more  of  the  maximum  pcssoole  rount 
resulted  in  a  correct  (R/R  or  S/S)  forecast. 

Ability  to  distinguish  between  predominately  intermittent 
and  pr edom i na te ly  steady  precipitation  appears  very  good. 
Seventy  percent  of  all  verifiable  forecasts  were  correct  and  only 
one  area/Tau  set  was  less  than  65%  correct  (ME  ATL,  T123  was 


•  'I 


vl 


I 


► 
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he  difference  is  probably  not  significant 


2. 2. 3. 2. 9  Frozen  Precipitation  -  yes/no .  The  results  of  t h 
verification  are  presented  in  Table  15. 

TABLE  15.  Frozen  Precipitation  Forecasting  Skill. 

FORECAST/VERIFICATION  COUNTS 
FIVE  DAY  FCSTS  SEVEN  DAY  FCSTS  ALL  FORECAST 
LOCATION  Y/Y  N/N  Y/N  N/Y  Y/Y  N/N  Y/N  N/Y  Y/Y  N/N  Y/N  N 

Kodiak  -  6 13  1 

Adak  7  1 -  7 - 

Astoria  4 - 4 - 

All  NE  PAC  -  17  1 1  14  1 

Sable  Island  -  4 - 

Key  West  - - - 

Hatteras  1 - 

All  NW  ATL  -  5 - 


Valencia 

— 

5 - - 

-  5 - 

— 

— 

11 

Kef  la vi k 

1 

9 - 

8  ___ 

1 

1 

17 

La  Coruna 

— 

4  ___ - 

-  5 - 

— 

— 

9 

All  NE  ATL 

1 

19 - - 

-  13 - 

1 

1 

37 

TOTAL 

1 

41  1 - 

135  1 

1 

2 

76 

The  counts  in  this  verification  are  low  because  ea 
requires  both  the  forecast  and  the  reported  occurrence 
precipitation.  A  minor  modification  to  the  Evaluation  Plan 

scoring  rules  was  the  addition  of  WW  codes  22,  23  and  25 
frozen  indicators. 


As  with  25  knot  winds 
because  the  occurrence  of 


assessment 


,  skill 
the  event  at 
infrequent  during  the  forecasting  period, 
evaluation  is  required. 


the  srat 
A  winter 


i  m  p  c  5  s  i 
:  n  was 


2.2.3.2.10  Surface  Visibility  <3  Mi  -  yes/no.  The  results 
this  verification  are  presented  in  Table  16. 


TABLE  16.  Visibility  <3  Miles  Forecasting  Skill. 


FORECAST/VERIFICATION 

COUNTS 

FIVE 

DAY 

FCSTS 

SEVEN  DAY  FCSTS 

ALL 

FORECAS 

LOCATION' 

Y/Y 

N/N 

Y/N 

N/Y 

Y/Y 

N/N 

Y/N 

N/Y 

Y/Y 

N/N 

Y  '  N 

Kodiak 

— 

16 

— 

— 

— 

16 

— 

— 

— 

32 

— 

Adak 

— 

15 

— 

— 

— 

15 

— 

— 

— 

30 

— 

Astoria 

— 

15 

— 

— 

— 

15 

— 

1 

— 

30 

— 

All  NE  PAC 

— 

46 

— 

— 

— 

46 

— 

1 

— 

92 

— 

Saole  Island 

— 

9 

3 

4 

2 

7 

3 

3 

2 

16 

6 

Key  West 

— 

14 

— 

— 

— 

15 

— 

— 

— 

29 

— 

Hatteras 

— 

12 

2 

— 

— 

14 

— 

— 

— 

26 

2 

All  NW  ATL 

— 

35 

5 

4 

2 

36 

3 

3 

2 

71 

8 

Valencia 

— 

15 

1 

— 

— 

14 

1 

1 

_ 

29 

2 

Kef lavik 

1 

12 

1 

2 

— 

13 

— 

3 

1 

25 

1 

La  Coruna 

— 

15 

— 

1 

— 

15 

— 

1 

— 

30 

— 

All  NE  ATL 

1 

42 

2 

3 

— 

42 

1 

5 

1 

84 

3 

TOTAL 

1 

123 

7 

7 

2 

124 

4 

9 

3 

247 

11 

Again, 

the 

occurrence 

of  the 

event 

was  too 

rare 

meaningful  skill  assessment. 


2. 2. 3. 3  Intertropical  Covergence  Zone  (ITCZ)  Location, 
indicated  in  the  Evaluation  Plan,  the  TAU  120  and  168  wind  fie 


at  1330  and  353  mb  plotted  as  wind  cards  were  ::  be  used  t  • 
forecast  the  development  or  movement  of  the  ITCZ.  Satellite 
imagery  (both  visual  and  IR)  was  to  be  used  for  ver : float:  :  n  . 

During  a  one  week  warm-up  and  for  the  first  fore  oast 
slightest  confluence  in  the  forecast  winds  was  used  to  forecast 
the  formation  of  an  ITCZ.  However,  when  verification  did  not 
support  these  forecasts,  a  more  conservative  evaluation  of  t.oe 
forecast  fields  was  used  for  the  remaining  6-7  weeks  of  the 
evaluation  period.  Though  the  ITCZ  forecaster  "knew  from 
experience"  that  the  ITCZ  should  form  very  soon,  he  nevertheless 
followed  the  NOGAPS  forecast  field  guidance  which,  wire 
conservative  evaluation,  never  indicated  the  formation  of  an  ITCZ 
in  either  hemisphere.  3ased  upon  the  available  satellite  data, 
these  no-ITCZ  forecasts  were  all  correct.  The  NEPRF  Contract 
Monitor  was  informed  after  a  few  weeks  of  the  persistent  lack  of 
any  verifiable  ITCZ.  It  was  decided  to  continue  forecasting, 
however,  since  it  was  reasonable  to  expect  the  formation  of  a 
convergence  zone  sometime  during  the  remainder  of  the  evaluation 
period . 

The  wind  shear  lines  penetrating  into  the  tropics  from  the 
poleward  regions  were  well  reflected  in  both  the  analysis  and 
forecast  fields.  The  observed  cloudiness  around  low  pressure 
systems  was  also  well  depicted  by  the  convergence  in  both  the 
analysis  and  forecast  fields.  But,  from  the  forecast  fields  it 
was  also  apparent  that  no  ITCZ  would  form  and,  as  previously 
stated,  none  did  form  during  the  entire  eight- week  period. 
Therefore,  whether  the  forecast  (or  analysis)  wind  field  would 
properly  portray  a  developing  or  existing  ITCZ  could  not  be 
determined  during  this  evaluation. 

The  satellite  imagery  used  for  verification  was  the  DMSP  (as 
processed  by  FNOC)  and  the  "Japanese  GOES"  as  received  by  FNOC. 
Verification  of  the  analyses  and  forecasts  were  easier  with  the 
Japanese  satellite  data;  but,  since  it  was  common  far  at  least 
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r  a 


c  a  a  c  a 


one  "panel" 
of  receipt, 


the 


composite  GOES  image  to  be  missing 
0 M 5 ?  data  were  also  required  for  ver 


2.2. 3.4  Sensible  Weather  Forecasting  Results  Summary.  Of  the  11 
sensible  weather  forecast  parameters  (including  the  ITCZ),  skill 
was  snown  only  in  one  (rain  or  shower).  Four  were  inconclusive 
due  to  insufficient  events  at  the  stations  during  the  evaluation 
period  (wind  >_  25  kt,  frozen  precipitation,  visibility 
restriction  and  ITCZ  location).  No  skill  was  demonstrated  in 
forecasting  the  remaining  six  parameters  -  surface  wind  speed, 
surface  wind  direction,  surface  air  temperature,  average 
cloudiness,  lowest  cloud  base  and  precipitation  expected. 


2. 2. 3. 5  Forecaster  Comment  Summary.  The  second  page  of  the 
forecaster  questionnaire  (Figure  2-12)  was  filled  out  following 
the  eight-week  forecasting  period.  Comments  received  were  highly 
diverse,  but  opinions  shared  by  at  least  two  of  the  four 
forecasters  follow. 


Particularly  Useful  NOGAPS  Products:  500  mb  heights, 
surface  pressure  and  air  temperature  and  1003  mb  winds  for 
station  forecasting;  1000  mb  and  850  mb  winds  for  ITCZ 
forecasting.  Reasons  were  generally  not  provided. 

Least  Useful  NOGAPS  Products :  PBL  depth,  stratus  thickness 
and  stratus  frequency;  because  of  unfamiliarity  and 
difficulty  reading  the  plots. 

Recommended  Product  Changes  o r  Additions:  No  consensus  at 
all . 


Recommended  Procedura  1  Changes :  Make  some  sort  of  running 
(or  recent)  score  (or  model  skill)  assessment  (statistics  or 
display)  available  to  the  forecasters. 


MRFE  FORECASTER  QUESTIONS I  RE  (continued) 


NAME:  _  DATE: _ 

WHAT  NOGAPS  PRODUCTS  WHERE  PARTICULARLY  USEFUL  AS  GUIDANCE  AND 
WHY? _ 


WHAT  NOGAPS  PRODUCTS  WERE  NOT  USEFUL  AS  GUIDANCE  AND  WHY’ 


WHAT  PRODUCT  CHANGES  OR  ADDITIONS  DO  YOU  RECOMMEND  FOR  FUTURE 
EVALUATIONS’  _ 


WHAT  PROCEDURAL  CHANGES  DO  YOU  RECOMMEND  FOR  FUTURE  EVALUATIONS? 


2. 2. 4.1  Evaluation  Procedure.  A 


the  Evaluation  Plan,  SEI3  tracks  up  to  5  low  pressure  centers  and 
correlates  analyzed  center  positions,  shapes  and  intensities  with 
forecast  centers.  Errors  related  to  the  forecast  track  and 


Errors  related  to  the  forecast 


.rack  and 


center  amplitude  are  calculated.  For  this  evaluation,  the 
operational  application  of  SEIS  over  the  North  Pacific  at  FNGC 
for  TAU's  00  through  60  was  modified  and  extended  to  permit 
calculation  of  TAG  72-163  errors  over  the  same  North  Pacific  area 
(see  Figure  2-02).  Only  00  GMT  forecast  runs  were  evaluated 
since  longer-range  forecasts  were  not  prepared  based  on  the  12 
GMT  observed  data.  Intermediate  12  hourly  analyzed  track 
continuity  was  maintained,  however. 


SEIS  data  collection  for  this  evaluation  began  on  10  April 
and  continued  through  15  June,  but  because  of  several  problems 
only  30  SEIS  Vortex  Tracking  runs  were  successfully  completed 
through  TAG  163.  3 e cause  the  three  days  of  data  collected  on  12, 
14  and  15  June  was  separated  from  earlier  data  by  two  weeks  and 
therefore  lacked  track  continuity,  it  was  decided  to  reduce  only 
that  data  collected  between  10  April  and  28  May. 

Statistical  summaries  were  produced  with  newly  revised  NEPRF 
software  which  did  not  yet  include  track  error  and  timing  error 


calculations. 


These  are  therefore  not  included  in  the 


but  some  related  qualitative  conclusions  are  reached  using 
latitudinal  and  longitudinal  error  statistics. 


2 . 2 . 4 , 2  Resu Its .  NOGAPS  success  in  forecasting  the  life  cycle 
(not  the  precise  track)  of  27  individual  North  Pacific  storms 
tracked  by  SEIS  during  the  49-day  period  is  summarized  in  Table 


JEI5  Tr3c.cn:  ;mar' 


TAUs 

30  24  43  72  96  123  144  163 


Nr.  of  low  pressure 
centers  analyzed 

(TAU  00)  or  forecast:  112  92  100  87  75  60  60  68 

Nr.  of  centers  matched 
and  verified  by  3EIS 

at  indicated  TAU:  59  43  33  24  17  11  9 

Nr.  of  centers  matched 
and  verified  at  some 

other  TAU:  17  29  29  28  21  23  20 

Nr.  of  centers  never 

matched  by  SEI3:  16  28  28  23  22  26  39 

The  top  line  of  the  table  shows  that  112  low  pressure 
centers  were  identified  and  tracked  by  SEIS,  87  substantial 
storms  (by  SEIS  selection  criteria)  were  forecast  by  the 
prediction  model  to  exist  at  TAU  72  and  68  were  predicted  for  TAU 
163  (day  7  of  a  forecast  cycle) .  Since  there  were  27  storms 
tracked  and  112  centers  analyzed,  the  average  storms  life  was 
approximately  four  days.  The  general  decrease  in  the  number  of 
low  centers  with  increasing  TAU  probably  reflects  a  loss  of 
ability  to  forecast  the  shorter  waves.  The  unexplained  increase 
to  68  at  T 16 8  would  be  significant  if  it  was  caused  by  increased 
noise  in  the  model. 

The  second  line  shows,  however,  that  only  30  of  the  37 
three-day  forecast  centers  verified  on  that  day  and  only  9  of  the 
68  seven-day  forecast  centers  were  actually  verified  by  SEIS. 
That  is  not  to  say  that  the  other  59  seven  day  forecast  centers 


57 


did  not  exist,  per  naps  very  near  t.nei 
they  did  not  have  SEIS  track  ccntinuitv 


o  o  s  1 1 1  o  r 


;ntmuitv  to  this  o  o  i  n  t . 


Line  three  shows  the  large  number  of  forecast  centers  which 
verified  at  one  or  more  other  TAU ' s  but  which  were  not  verifiable 
by  SEIS  at  the  indicated  TAU.  In  most  cases  the  actual  low  will 
have  weakened  below  the  SEIS  tracking  threshold  at  an  earlier 
TAU.  If  the  low  subsequently  reintensified  SEIS  would  consider 
that  a  case  of  cyclogenesis,  the  center  would  be  given  a  new 
identifier,  and  SEIS  would  look  for  a  center  match  with  a  new 
forecast  low  (not  with  an  older  continuously  tracking  forecast 
low) .  In  5  of  the  27  cases  the  low  was  predicted  by  MOGAPS  to 
form  earlier  than  it  was  actually  picked  up  on  the  analysis  by 
SEIS,  but  in  22  cases  (not  necessarily  the  other  22  cases)  the 
SEIS-tracked  lows  were  predicted  to  retain  their  identify  longer 
than  observed.  Cyclogenesis  was  forecast  in  advance  in  only  8  of 
the  27  cases  and  never  consistently  in  a  series  more  than  2  days 
in  advance.  Line  four  centers  are  clear  false  alarms  -  those 
cases  where  SEIS  never  matched  a  forecast  center  track  with  an 
analyzed  center  track.  Of  87  centers  forecast  to  exist  at  TAU 
72,  28  (22%)  never  formed  or  at  least  never  formed  where  they 
could  be  correlated  with  an  analyzed  storm  track.  At  TAU  153 
this  percentage  increases  to  57. 


Table  13  provides  error  statistics  (forecast  minus  analysis' 
for  those  forecast  surface  lows  which  were  successfully  matched 
with  analyzed  lows  and  verified  by  SEIS  (those  identified  in  line 
2  of  Table  17 ) . 


..iciLL  .a.  cj-*o  irr:r  summary. 


TAJ  5 


24 

43  72 

96 

123 

144 

16  3 

Pressure 

(mb)  : 

Avg 

Error 

-1.7 

-3.7  -2.5 

-3.7 

-4.9 

1.  5 

2.  1 

STD 

of  Error 

4.32 

6.59  7.67 

9.51 

11.  35 

11.87 

9.29 

La  t  i  tude 

(deg)  : 

Avg 

Error 

0.0 

-1.2  -1.6 

-0.8 

0.  5 

0. 4 

2.3 

STD 

of  Error 

2.35 

2.69  4.22 

3.65 

3.46 

3.  35 

3.67 

Longitude 

(deg) 

:  Avg 

Error 

-0.  5 

3.3  -0.7 

-1.6 

-1.2 

-2.9 

-3  .  0 

STD 

of  Error 

5.41 

4.66  7.03 

6.49 

6.32 

9.  36 

12.33 

Distance 

( nm)  : 

Avg 

Error 

177 

214  316 

304 

285 

350 

477 

STD 

of  Error 

180 

128  204 

142 

153 

212 

233 

Nr.  of  Cases: 

59 

43  30 

24 

17 

11 

9 

Ther 

e  is 

an  . 

abrupt  shift  between  TAU' 

3  123 

and 

14  4 

from 

forecasting  too  much  amplitude  (pressure  centers  too  low)  to 
their  being  too  high  in  central  pressure.  Distance  errors 
increase  markedly  as  would  be  expected  at  the  higher  TAU  '  s  and 
are  increasingly  biased  toward  the  east  (Note:  the  longitude 
convention  is  0-363  westward)  and  shift  from  south  of  track  at 
lower  TAU ' s  to  north  of  track  at  higher  TAU's.  The  model  is 
obviously  fast  in  movement  and  to  the  right  of  the  actual  track. 


SECTION  3 


SUMMARY  CONCLUSIONS  AND  RECOMMENDATIONS 


3.1  ?r::ed-res  .  Except  for  the  proolem.s  when  os:  no  ma  ;  r.  e  t :  : 
tape  for  data  scores,  the  procedures  recurred  to  implement  toe 
Medium-Ranee  Forecast  System  Evaluation  Plan  worked  very  well  and 
are  repeatable  by  NEPRF.  A  detailed  description  of  the  magnet:: 
tape  problems  are  delineated  in  SASC,  1984.  All  job  streams  and 
corrected  software  developed  during  this  contract  have  been 
delivered  to  the  Contract  Monitor.  All  suggested  changes  to 
these  procedures  are  contained  in  subsection  3.4  of  this  report. 

3.2  Accuracy  Assessment.  In  this  subsection  several  key 
measures  of  forecast  system  skill  are  displayed  in  measure  value 
versus  TAU  graphic  summary  form. 

Figure  3-01  (bottom)  shows  the  anomaly  correlation  (AC) 
scores  at  the  four  levels  for  the  four  TAUs.  The  European  Centre 
for  Medium  Range  Weather  Forecasts  (SCMWF)  has  concluded  that  an 
AC  score  must  be  at  least  .50  to  be  useful.  If  such  a  criteria 
is  reasonable  for  Navy  applications,  one  sees  little  skill  past 
T96  at  1000  mb,  little  past  T120  at  850  mb  and  little  past  T144 
at  the  two  highest  levels. 

Figure  3-01  (top)  shows  the  considerable  greater  skill  in 
the  lower  wave  numbers  (WN)  at  500  mb  over  the  Northern 
Hemisphere  and  also  how  the  difference  between  WN  1-3  and  WN  4-9 
increases  dramatically  at  the  higher  TAUs. 

Figure  3-02  illustrates  the  difference  in  skill  from  area  to 
area  at  500  mb  (top)  and  at  1000  mb  (bottom) .  The  ocean  areas 
show  more  skill  than  the  hemispheres  at  1000  mb,  but  this  does 
not  hold  true  at  500  mb  in  the  Northern  Hemisphere. 

In  Figure  3-03  we  see  the  month- to-month  variation  between 
areas.  This  is  particularly  marked  in  the  South  HEMI  which  snows 
much  more  skill  in  April  than  in  May. 


ins  .-.w  scores  ror  temperature  are  snov/n  m  rigure 
Unlike  the  heights  (Figure  3-01  (bottom)),  the  temperatures  show 
greater  skill  at  the  lower  levels,  particularly  at  the  longer 
TAUs . 


RMS  errors  of  wind  are  depicted  in  Figure  3-05.  The  scalar 
errors  (top)  increase  only  slightly  as  TAU  increases  and  are 
generally  proportional  to  the  mean  wind  at  the  four  levels.  The 
directional  errors  (bottom)  also  increase  slightly  with 
increasing  TAU,  but  are  about  20  degrees  larger  at  the  lower  two 
levels . 

Figure  3-06  compares  the  two  month  combined  Northern 
Hemispheric  mean  height  errors  at  the  four  levels  (top)  with  the 
mean  temperature  errors  (bottom).  The  same  information  for  the 
Southern  Hemisphere  is  shown  in  Figure  3-07.  Note  the  warming 
with  time  at  200  mb  in  both  hemispheres  and  the  corresponding 
large  positive  height  errors  -  particularly  in  the  Southern 
Hemisphere.  This  same  warming  tendency  is  also  noticeable  an  500 
mb  in  the  Southern  Hemisphere.  Cooling  is  noted  at  the  lower 
levels  in  both  hemispheres. 

3.3  Utility  Assessment.  The  Evaluation  Plan  stated  that  the 
forecast  system's  usefulness  -  its  ability  to  meet  the  needs  of 
operational  forecasters  -  would  be  assessed.  Such  an  assessment, 
however,  becomes  difficult  and  potentially  misleading  when  the 
data  collection  phase  is  as  short  as  this  one  was  and  the  number 
of  significant  phenomenological  events  are  so  limited.  In  the 
interest  of  providing  an  outline  for  future  evaluations,  a 
minimal  assessment  is  provided  below;  but  the  reader  is  cautioned 
to  not  make  any  value  judgments  from  what  follows.  A  true 
utility  assessment  should  consider  the  recommendations  made  in 
subsection  3.4  and  will  certainly  require  a  longer  period  of  data 
col  lect  i  on . 


Certain  accuracy  criteria  were  set  f o  r  tn  m  ■ .soerry  et  a  - 
(1934)  and  are  reproduced  here  as  Figure  3-33.  These  criteria 
provide  a  potential  measure  of  usefulness.  In  discussing  them 
below  we  will  consider  only  the  day  five  results,  since  the 
evaluation  forecasts  extended  no  further  than  day  seven. 

Extratropical  S  torm  Track :  Referring  to  the  tables  in 
subsection  2. 2. 4. 2,  latitudinal  error  of  less  than  1.0  deg  (63 
nm)  should  approximate  the  average  track  error  for  eastward 
moving  storms  and  the  latitudinal  error  of  less  than  1.5  deg  (93 
nm)  should  approximate  the  track  error  for  northward  moving 
storms.  Since  both  are  well  within  the  200  nm  criteria  of  Figure 
3-31,  these  results  are  encouraging  as  is  a  latitudinal  3D  of 
only  3.46  deg  (208  nm) .  Less  encouraging  is  the  longitudinal  SD 
of  6.82  deg  (409  nm) .  And  of  course  these  figures  do  not  include 
any  errors  for  the  22  centers  forecast  to  exist  at  TAU  120  but 
never  matched  by  SEIS. 

Surface  Wind  Speed:  Based  on  the  sensible  weather 
forecasting  results  (subsection  2. 2. 3. 2. 2),  the  5.6  kt  error  ST 
is  far  more  than  25  percent  of  the  observed  9.0  kt  average. 
Observed  1300  mb  values  are  not  available  in  the  Appendix,  but 
RMS  errors  at  1000  mb  of  8-10  kt  must  surely  be  more  than  25%  of 
the  actual  April-May  mean  low  level  wind  speed. 

Surface  Wind  Direction:  Sensible  weather  forecasters  were 
able  to  predict  day-five  wind  directions  within  45  deg  only  30% 
of  the  time.  This  is  not  surprising  since  RMS  directional  errors 
at  1000  mb  were  all  in  excess  of  70  deg. 

Free  A tmosphe r i c  Wind  Speed:  Without  observed  mean  values, 
it  is  impossible  to  determine  if  the  20  percent  criteria  was  met. 

Free  Atmospheric  W i nd  Direction:  RMS  wind  direction  errors 
approach  45  deg  at  200  mb  in  the  S.  HEMI  (47  deg  at  T120)  but  at 
all  other  levels  and  in  all  other  areas  they  are  higher;  as  much 
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cumuli form/mixed/ 
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Surface  Temperatures :  Based  on  sensible  weather  forecasting 
results  (subsection  2. 2. 3. 2. 4),  the  5.3  deg  error  SB  is  53 
percent  of  the  9.3  observed  SD  (far  more  than  the  43  percent 
criteria).  Observed  1003  mb  temperature  values  are  not  available 
from  the  field  verifications. 

Free  A tmosphere  Temperatures :  RMS  temperature  errors  at  day 
five,  T120  are  all  very  close  to  5  deg  -  somewhat  less  at  853  and 
530  mb,  very  slightly  more  at  200  mb. 

Cloud  Cover:  3ased  on  sensible  weather  forecasting  results 
(subsection  2. 2. 3. 2. 5),  an  error  SD  of  3.2  eighths  is  over  50%  of 
observed  5.6  eighths  mean  and  even  greater  than  the  2.6  eighths 
SD  of  the  observations. 

Cloud  Type  (dominant) :  There  was  no  attempt  to  verify  the 
models  ability  to  forecast  types  of  clouds. 

C  loud  Base  (dominant) :  Based  on  sensible  weather 
forecasting  results  (subsection  2. 2. 3. 2. 6),  the  error  SD  of  3433 
ft  was  greater  than  both  the  observed  mean  and  the  observed  SD 
and  was  43  percent  of  the  maximum  possible  range  (0-8030  ft). 

Precipitation  E  xpected:  Based  on  sensible  weather 
forecasting  results  (subsection  2. 2. 3. 2. 7),  the  models  guidance 
value  for  precip i ta t ion-yes-o r-no  predictions  was  negligible. 

P rec ipitation  Amount :  There  was  no  attempt  to  verify  the 
models  ability  to  forecast  amount  of  precipitation. 

Precipitation  Type :  Based  on  sensible  weather  forecasting 
results  (subsection  2. 2. 3. 2. 8),  the  ability  to  distinguish 
between  rain  and  showers  (when  they  occurred)  was  good  -  ~3 
percent  of  such  verifiable  forecasts  were  correct. 


Rrecioitation 


:ozen:  mere  were  msurri; 


ioitation  events  observed  or  forecast  durina  this  evaluation. 


Visibility  Restriction: 


There  were  insufficient 


observations  or  forecasts  of  restricted  visibility  during  this 
evaluation. 


3.4  Recommendations  for  Future  Evaluations. 


In  this  subsection 


changes  which  would  improve  similar  future  evaluations  are  listed 
in  the  same  general  order  as  the  findings  were  presented  in 
Section  2.  But  first  two  general  recommendations: 

•  Conduct  the  evaluation  over  a  longer  period  which,  as  a 
minimum,  would  include  a  complete  winter  season.  This  would 
provide  sufficient  gales,  mature  storms  and  other  adverse 
weather  events  to  permit  meaningful  gale  warning,  SEI5  and 
sensible  weather  forecasting  evaluations. 

•  Prohibit  major  forecast  model  changes  during  the  course  of 
the  evaluation.  But,  if  such  changes  are  required,  the 
expected  ramifications  should  be  carefully  documented  at  the 
time  they  are  made  so  that  effects  can  be  monitored  by  and 
accounted  for  in  the  evaluation. 


Review  the  acceptable  levels  of  accuracy  set  forth  in  the 
Evaluation  Plan  (Figure  3-08  herein)  and  restate  in  more 
easily  interpreted  and  measurable  terms.  For  example:  In 
the  plan  acceptable  five  day  temperature  accuracy  is 
specified  as  seasonal  standard  deviation  times  0.4.  This 
could  be  more  precisely  stated  as  "standard  deviation  of  the 


error  <  seasonal  observed  standard  deviation 


0.4". 


Similar  measures  should  be  considered  for  wind  speed  and 
cloud  cover.  Also,  acceptable  storm  track  accuracy  would  be 
better  expressed  in  terms  of  standard  deviation  of  the  error 
rather  than  average  error. 


.a-culate  ooser  vec 


(TAu  33}  means  and  stancar; 


and  make  these  available  for  inclusion  in  an  appendix  an 
for  analytical  purposes. 

Eliminate  calculating  the  standard  deviation  of  th 
verifying  anomaly  since  these  statistics  have  no  apparen 
value  in  the  assessment  of  medium-range  forecasting  skill. 


Eliminate  the  common  longitude  boundary  at  133  in  z h 
Pacific  and  at  35W  in  the  Atlantic.  This  would  preclud 
overlap  which  causes  the  same  data  to  enter  separate  subate 
statistics . 


Special  Verifications. 


•  Add  track  error  and  timing  error  calculations 
SEIS  statistical  package. 


f-  -  ua 


•  Establish  another  mean  storm  track  regional  grid  over  th 
North  Atlantic  and/or  other  a rea(s)  of  Navy  interest  so  tha 
skill  can  be  compared  between  regions. 


Distribute  mean  storm  track  plots 


several 


meteorologists  for  their  evaluation  as  forecasting  guidance 
Their  opinions  as  to  the  relative  day-to-day  worth  of  th 
plots  should  also  be  solicited  and  used  to  determine 
threshold  of  usefulness  for  correlation  coef f  i  c i en ts . 


Eliminate  the  over ly- i n vo 1 ved  30-36  kt  gale  warning  scorin 
criteria,  but  retain  the  percent  scores  for  grid-point  coun 
differences,  thus  providing  a  measure  of  to  what  degre 
gales  are  under-  or  over-forecast. 


of  degrees,  as  reported,  rather  than  by  quadrants 
octants.  This  would  be  easier  to  score  and  more  precise 

Involve  currently  active  Navy  forecasters  -  at  Ocean  Cen 
and/or  Detachments  -  in  sensible  weather  forecasting. 

Discontinue  or  else  redraft  the  sensible  weather  forecas 
questionnaire  in  more  precise  yes/no,  multiple  choice  te 

Provide  sensible  weather  forecasters  some  skill  feed- 
such  as  their  raw  scores  and/or  field  verification  sc 
for  their  forecasting  areas/points  during  the  course  of 
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ATL 


FNOC 


HEM  I 

mb 


MRFE 


NEPRF 


NOGAPS 


PAG 


RMS 


SD 

3  E I  S 


TAu 


TROP 


anomaly  cc  r  r e  1  a  t :  :  r. 

Atlantic  Ocean 

Fleet  Numerical  Oceanography  Center 
Hemisphere  (north  or  souc.o 
millibar  (unit  of  ac.mcspher  :  ;  press  -  re 
Medium-Range  Forecast  Evaluation 

Naval  Environmental  Prediction  Research  Facil: 

Navy  Operational  Global  Atmospheric  Predio 
System 

Pacific  Ocean 

root-mean-square  (error) 

standard  deviation 

Sy s tema tic  Error  Identification  System 
Same  as  TAU  (see  below) 
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length  of  forecast  in  hours 
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